v7.4.0 release notes

pdfinfo: tolerate PDFs that overflow and underflow the graphics stack
Require pikepdf 0.9.1
2018-12-15 15:27:23 -08:00 · 2018-12-15 15:10:29 -08:00 · 2018-12-15 14:23:10 -08:00 · 2018-12-14 23:29:26 -08:00 · 2018-12-14 23:21:13 -08:00 · 2018-12-14 23:06:57 -08:00
19 changed files with 312 additions and 244 deletions
@@ -0,0 +1,19 @@
+# OCRmyPDF webservice
+#
+FROM      jbarlow83/ocrmypdf-polyglot:latest
+
+USER root
+
+# Update system and install our dependencies
+RUN apt-get update && apt-get install -y --no-install-recommends \
+  python3-flask
+
+RUN apt-get autoremove -y && apt-get clean -y
+
+EXPOSE 5000
+
+COPY .docker/webservice.py /application
+
+USER docker
+
+ENTRYPOINT ["python3", "/application/webservice.py"]
@@ -0,0 +1,94 @@
+# webservice.py wrapper for OCRmyPDF
+# Copyright (C) 2018 James R. Barlow: github.com/jbarlow83
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+"""This is a simple web service/HTTP wrapper for OCRmyPDF
+
+This may be more convenient than the command line tool for some Docker users.
+Note that OCRmyPDF uses Ghostscript, which is licensed under AGPL3+. While
+OCRmyPDF is under GPL3, this file is distributed under the Affero GPL3+ license,
+to emphasize that SaaS deployments should make sure they comply with
+Ghostscript's license as well as OCRmyPDF's.
+"""
+
+from flask import Flask, Response, flash, request, redirect, url_for, abort, send_from_directory
+from subprocess import run, PIPE
+from tempfile import TemporaryDirectory
+from werkzeug.utils import secure_filename
+import os
+import shlex
+
+app = Flask(__name__)
+app.secret_key = "secret"
+
+uploaddir = TemporaryDirectory(prefix="ocrmypdf-upload")
+downloaddir = TemporaryDirectory(prefix="ocrmypdf-download")
+
+app.config["UPLOAD_FOLDER"] = uploaddir
+ALLOWED_EXTENSIONS = set(["pdf"])
+
+
+def allowed_file(filename):
+    return "." in filename and filename.rsplit(".", 1)[1].lower() in ALLOWED_EXTENSIONS
+
+
+def do_ocrmypdf(file):
+    filename = secure_filename(file.filename)
+    up_file = os.path.join(uploaddir.name, filename)
+    file.save(up_file)
+    down_file = os.path.join(downloaddir.name, filename)
+
+    cmd_args = [arg for arg in shlex.split(request.form["params"])]
+    if "--sidecar" in cmd_args:
+        return Response("--sidecar not supported", 501, mimetype='text/plain')
+
+    ocrmypdf_args = ["ocrmypdf", *cmd_args, up_file, down_file]
+    proc = run(ocrmypdf_args, stdout=PIPE, stderr=PIPE, encoding="utf-8")
+    if proc.returncode != 0:
+        stderr = proc.stderr
+        return Response(stderr, 400, mimetype='text/plain')
+
+    return send_from_directory(downloaddir.name, filename)
+
+
+@app.route("/", methods=["GET", "POST"])
+def upload_file():
+    if request.method == "POST":
+        if "file" not in request.files:
+            return Response("No file in POST", 400, mimetype='text/plain')
+        file = request.files["file"]
+        if file.filename == "":
+            return Response("Empty filename", 400, mimetype='text/plain')
+        if not allowed_file(file.filename):
+            return Response("Invalid filename", 400, mimetype='text/plain')
+        if file and allowed_file(file.filename):
+            return do_ocrmypdf(file)
+        return Response("Some other problem", 400, mimetype='text/plain')
+
+    return """
+    <!doctype html>
+    <title>OCRmyPDF webapp</title>
+    <h1>Upload a PDF (debug UI)</h1>
+    <form method=post enctype=multipart/form-data>
+      <label for="args">Command line parameters</label>
+      <input type=textbox name=params>
+      <label for="file">File to upload</label>
+      <input type=file name=file>
+      <input type=submit value=Upload>
+    </form>
+    """
+
+if __name__ == "__main__":
+    app.run(host='0.0.0.0')
@@ -74,7 +74,7 @@ if on_rtd:
        def __getattr__(cls, name):
            return MagicMock()

-    MOCK_MODULES = ['pikepdf', 'libxmp', 'libxmp.utils', 'ocrmypdf.leptonica']
+    MOCK_MODULES = ['pikepdf', 'ocrmypdf.leptonica']
    sys.modules.update((mod_name, Mock()) for mod_name in MOCK_MODULES)


@@ -129,7 +129,7 @@ OCRmyPDF perform some image processing on each page of a PDF, if desired.  The s

 * ``--clean-final`` uses unpaper to clean up pages before OCR and inserts the page into the final output.  You will want to review each page to ensure that unpaper did not remove something important.

-* ``-mask-barcodes`` will "cover up" any barcodes detected in the image of a page. Barcodes are known to confuse Tesseract OCR and interfere with the recognition of text on the same baseline as a barcode. The output file will contain the unaltered image of the barcode.
+* ``--mask-barcodes`` will "cover up" any barcodes detected in the image of a page. Barcodes are known to confuse Tesseract OCR and interfere with the recognition of text on the same baseline as a barcode. The output file will contain the unaltered image of the barcode.

 .. note::

@@ -13,6 +13,22 @@ Note that it is licensed under GPLv3, so scripts that ``import ocrmypdf`` and ar
   find:    [^`]\#([0-9]{1,3})[^0-9]
   replace: `#$1 <https://github.com/jbarlow83/OCRmyPDF/issues/$1>`_

+v7.4.0
+------
+
+-   ``--force-ocr`` may now be used with the new ``--threshold`` and ``--mask-barcodes`` features
+
+-   pikepdf >= 0.9.1 is now required.
+
+-   Changed metadata handling to pikepdf 0.9.1. As a result, metadata handling of non-ASCII characters in Ghostscript 9.25 or later is fixed.
+
+-   chardet >= 3.0.4 is temporarily listed as required. pdfminer.six depends on it, but the most recent release does not specify this requirement. (`#326 <https://github.com/jbarlow83/OCRmyPDF/issues/326>`_)
+
+-   python-xmp-toolkit and libexempi are no longer required.
+
+-   A new Docker image is now being provided for users who wish to access OCRmyPDF over a simple HTTP interface, instead of the command line.
+
+-   Increase tolerance of PDFs that overflow or underflow the PDF graphics stack. (`#325 <https://github.com/jbarlow83/OCRmyPDF/issues/325>`_)

 v7.3.1
 ------
@@ -5,7 +5,7 @@ chardet == 3.0.4
 cffi == 1.11.5
 img2pdf == 0.3.1
 pdfminer.six == 20181108
-pikepdf == 0.3.7
+pikepdf == 0.9.1
 Pillow >= 5.0.0, != 5.1.0 ; sys_platform == "darwin"
 pycparser == 2.19
 python-xmp-toolkit == 2.0.1
@@ -11,6 +11,8 @@ ignore =
 [tool:pytest]
 norecursedirs = lib .pc .git output cache resources
 testpaths = tests
+filterwarnings =
+	ignore:.*XMLParser.*:DeprecationWarning

 [metadata]
-license_file = LICENSE
+license_file = LICENSE
@@ -248,14 +248,14 @@ setup(
        'src/ocrmypdf/lib/compile_leptonica.py:ffibuilder'
    ],
    install_requires=[
+        'chardet >= 3.0.4, < 4',  # unlisted requirement of pdfminer.six 20181108
        'cffi >= 1.9.1',          # must be a setup and install requirement
        'img2pdf >= 0.3.0, < 0.4',       # pure Python, so track HEAD closely
        'pdfminer.six == 20181108',
-        'pikepdf >= 0.3.7, < 0.4',
+        'pikepdf >= 0.9.1',
        'Pillow >= 4.0.0, != 5.1.0 ; sys_platform == "darwin"',
                                  # Pillow < 4 has BytesIO/TIFF bug w/img2pdf 0.2.3
                                  # block 5.1.0, broken wheels
-        'python-xmp-toolkit >= 2, < 3',
        'reportlab >= 3.3.0',     # oldest released version with sane image handling
        'ruffus >= 2.7.0',
    ],
@@ -545,11 +545,6 @@ def check_options_ocr_behavior(options, log):
        raise argparse.ArgumentError(
            None,
            "Error: choose only one of --force-ocr, --skip-text, --redo-ocr.")
-    if options.force_ocr and any((options.mask_barcodes, options.threshold)):
-        raise argparse.ArgumentError(
-            '--force-ocr',
-            'Error: --force-ocr currently may not be used with --threshold or --mask-barcodes'
-        )


 def check_options_optimizing(options, log):
@@ -29,9 +29,11 @@ import pikepdf
 from PIL import Image
 from ruffus import formatter, regex, Pipeline, suffix

+from pikepdf.models.metadata import encode_pdf_date
+
 from .hocrtransform import HocrTransform
 from .pdfinfo import PdfInfo, Colorspace
-from .pdfa import generate_pdfa_ps, encode_pdf_date
+from .pdfa import generate_pdfa_ps
 from .helpers import re_symlink, is_iterable_notstr, page_number, flatten_groups
 from .exec import ghostscript, tesseract
 from .exceptions import UnsupportedImageFormatError, \
@@ -598,10 +600,6 @@ def select_ocr_image(
    options = context.get_options()
    pageinfo = get_pageinfo(image, context)

-    if options.force_ocr:
-        re_symlink(image, output_file, log)
-        return
-
    with Image.open(image) as im:
        from PIL import ImageColor
        from PIL import ImageDraw
@@ -613,24 +611,27 @@ def select_ocr_image(
        xres, yres = im.info['dpi']
        log.debug('resolution %r %r', xres, yres)

-        mask = None  # Exclude both visible and invisible text from OCR
-        if options.redo_ocr:
-            mask = True  # Mask visible text, but not invisible text
+        if not options.force_ocr:
+            # Do not mask text areas when forcing OCR, because we need to OCR
+            # all text areas
+            mask = None  # Exclude both visible and invisible text from OCR
+            if options.redo_ocr:
+                mask = True  # Mask visible text, but not invisible text

-        for textarea in pageinfo.get_textareas(visible=mask, corrupt=None):
-            # Calculate resolution based on the image size and page dimensions
-            # without regard whatever resolution is in pageinfo (may differ or
-            # be None)
-            bbox = [float(v) for v in textarea]
-            xscale, yscale = float(xres) / 72.0, float(yres) / 72.0
-            pixcoords = [bbox[0] * xscale,
-                         im.height - bbox[3] * yscale,
-                         bbox[2] * xscale,
-                         im.height - bbox[1] * yscale]
-            pixcoords = [int(round(c)) for c in pixcoords]
-            log.debug('blanking %r', pixcoords)
-            draw.rectangle(pixcoords, fill=white)
-            #draw.rectangle(pixcoords, outline=pink)
+            for textarea in pageinfo.get_textareas(visible=mask, corrupt=None):
+                # Calculate resolution based on the image size and page dimensions
+                # without regard whatever resolution is in pageinfo (may differ or
+                # be None)
+                bbox = [float(v) for v in textarea]
+                xscale, yscale = float(xres) / 72.0, float(yres) / 72.0
+                pixcoords = [bbox[0] * xscale,
+                            im.height - bbox[3] * yscale,
+                            bbox[2] * xscale,
+                            im.height - bbox[1] * yscale]
+                pixcoords = [int(round(c)) for c in pixcoords]
+                log.debug('blanking %r', pixcoords)
+                draw.rectangle(pixcoords, fill=white)
+                #draw.rectangle(pixcoords, outline=pink)

        if options.mask_barcodes or options.threshold:
            pix = leptonica.Pix.frompil(im)
@@ -794,10 +795,10 @@ def ocr_tesseract_textonly_pdf(
        log=log)


-def get_pdfmark(base_pdf, options):
+def get_docinfo(base_pdf, options):
    def from_document_info(key):
        try:
-            s = base_pdf.metadata[key]
+            s = base_pdf.docinfo[key]
            return str(s)
        except (KeyError, TypeError):
            return ''
@@ -839,21 +840,34 @@ def generate_postscript_stub(
        context):
    options = context.get_options()
    pdf = pikepdf.open(input_file)
-    pdfmark = get_pdfmark(pdf, options)
+    generate_pdfa_ps(output_file)

-    ascii_docinfo = False
-    if ghostscript.version() >= '9.24':
-        ascii_docinfo = True
-        try:
-            for v in pdfmark.values():
-                v.encode('ascii', errors='strict')
-        except UnicodeEncodeError:
-            log.warning(
-                "Ghostscript 9.24+ does not support Unicode strings in "
-                " metadata. These will be converted to ASCII if possible."
-            )

-    generate_pdfa_ps(output_file, pdfmark, ascii_docinfo=ascii_docinfo)
+def convert_to_pdfa(
+    input_files_groups,
+    output_file,
+    log,
+    context
+):
+    options = context.get_options()
+    input_pdfinfo = context.get_pdfinfo()
+
+    input_files = list(f for f in flatten_groups(input_files_groups))
+    layers_file = next(
+        (ii for ii in input_files if ii.endswith('layers.rendered.pdf')), None
+    )
+    ps = next(
+        (ii for ii in input_files if ii.endswith('.ps')), None
+    )
+    ghostscript.generate_pdfa(
+        pdf_version=input_pdfinfo.min_version,
+        pdf_pages=[layers_file, ps],
+        output_file=output_file,
+        compression=options.pdfa_image_compression,
+        log=log,
+        threads=options.jobs or 1,
+        pdfa_part=options.output_type[-1]  # is pdfa-1, pdfa-2, or pdfa-3
+    )


 def metadata_fixup(
@@ -870,33 +884,23 @@ def metadata_fixup(
    layers_file = next(
        (ii for ii in input_files if ii.endswith('layers.rendered.pdf')), None
    )
-    ps = next(
-        (ii for ii in input_files if ii.endswith('.ps')), None
+    pdfa = next(
+        (ii for ii in input_files if ii.endswith('pdfa.pdf')), None
    )
+    metadata = pikepdf.open(metadata_file)
+    docinfo = get_docinfo(metadata, options)

-    if options.output_type.startswith('pdfa'):
-        input_pdfinfo = context.get_pdfinfo()
-        ghostscript.generate_pdfa(
-            pdf_version=input_pdfinfo.min_version,
-            pdf_pages=[layers_file, ps],
-            output_file=output_file,
-            compression=options.pdfa_image_compression,
-            log=log,
-            threads=options.jobs or 1,
-            pdfa_part=options.output_type[-1]  # is pdfa-1, pdfa-2, or pdfa-3
-        )
-    else:
-        metadata = pikepdf.open(metadata_file)
-        pdfmark = get_pdfmark(metadata, options)
-        pdf = pikepdf.open(layers_file)
-        pdf.metadata = pdf.make_indirect(pikepdf.Dictionary(pdfmark))
-        try:
-            pdf.save(output_file, compress_streams=True,
-                     object_stream_mode=pikepdf.ObjectStreamMode.generate)
-        except AttributeError:
-            # pikepdf <= 0.3.4
-            pdf.save(output_file,
-                     stream_data_mode=pikepdf.StreamDataMode.compress)
+    working_file = pdfa if pdfa else layers_file
+
+    pdf = pikepdf.open(working_file)
+    with pdf.open_metadata() as meta:
+        meta.load_from_docinfo(docinfo, delete_missing=False)
+        # If xmp:CreateDate is missing, set it to the modify date to
+        # match Ghostscript, for consistency
+        if 'xmp:CreateDate' not in meta:
+            meta['xmp:CreateDate'] = meta.get('xmp:ModifyDate', '')
+    pdf.save(output_file, compress_streams=True,
+             object_stream_mode=pikepdf.ObjectStreamMode.generate)


 def optimize_pdf(
@@ -1119,7 +1123,7 @@ def build_pipeline(options, work_folder, log, context):
        extras=[log, context])
    task_weave_layers.graphviz(fillcolor='"#00cc66"')

-    # PDF/A
+    # PDF/A pdfmark
    task_generate_postscript_stub = main_pipeline.transform(
        task_func=generate_postscript_stub,
        input=task_repair_and_parse_pdf,
@@ -1128,11 +1132,21 @@ def build_pipeline(options, work_folder, log, context):
        extras=[log, context])
    task_generate_postscript_stub.active_if(options.output_type.startswith('pdfa'))

+    # PDF/A conversion
+    task_convert_to_pdfa = main_pipeline.merge(
+        task_func=convert_to_pdfa,
+        input=[task_generate_postscript_stub,
+               task_weave_layers],
+        output=os.path.join(work_folder, 'pdfa.pdf'),
+        extras=[log, context]
+    )
+    task_convert_to_pdfa.active_if(options.output_type.startswith('pdfa'))
+
    task_metadata_fixup = main_pipeline.merge(
        task_func=metadata_fixup,
        input=[task_repair_and_parse_pdf,
               task_weave_layers,
-               task_generate_postscript_stub],
+               task_convert_to_pdfa],
        output=os.path.join(work_folder, 'metafix.pdf'),
        extras=[log, context]
    )
@@ -15,7 +15,7 @@
 # You should have received a copy of the GNU General Public License
 # along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.

-from functools import partial
+from functools import partial, wraps
 from collections.abc import Iterable
 from contextlib import suppress
 from pathlib import Path
@@ -168,3 +168,16 @@ def flatten_groups(groups):
            yield from obj
        else:
            yield obj
+
+
+def deprecated(func):
+    """Warn that function is deprecated"""
+    @wraps(func)
+    def new_func(*args, **kwargs):
+        warnings.simplefilter('always', DeprecationWarning)  # turn off filter
+        warnings.warn("Call to deprecated function {}.".format(func.__name__),
+                      category=DeprecationWarning,
+                      stacklevel=2)
+        warnings.simplefilter('default', DeprecationWarning)  # reset filter
+        return func(*args, **kwargs)
+    return new_func
@@ -567,6 +567,8 @@ class Pix(LeptonicaObject):
            with _LeptonicaErrorTrap():
                pix = Pix(lept.pixConvertTo8(self._cdata, 0))
                pixa_candidates = PixArray(lept.pixExtractBarcodes(pix._cdata, 0))
+                with suppress(FileNotFoundError):
+                    os.unlink('junkpixt.png')  # leptonica may produce this
                sarray = StringArray(lept.pixReadBarcodes(
                    pixa_candidates._cdata,
                    lept.L_BF_ANY,
@@ -299,8 +299,9 @@ def convert_to_jbig2(pike, jbig2_groups, root, log, options):
            jbig2_im_data = jbig2_im_file.read_bytes()
            im_obj = pike.get_object(xref, 0)
            im_obj.write(
-                jbig2_im_data, pikepdf.Name('/JBIG2Decode'),
-                jbig2_globals_dict
+                jbig2_im_data,
+                filter=pikepdf.Name('/JBIG2Decode'),
+                decode_parms=jbig2_globals_dict
            )


@@ -32,14 +32,19 @@ Ghostscript's handling of pdfmark.
 """

 from binascii import hexlify
-from datetime import datetime
 from pathlib import Path
 from string import Template
 import pkg_resources
 import os

-from libxmp.utils import file_to_dict
-from libxmp import consts
+import pikepdf
+
+from pikepdf.models.metadata import (
+    encode_pdf_date as _encode_date,
+    decode_pdf_date as _decode_date
+)
+
+from .helpers import deprecated


 ICC_PROFILE_RELPATH = 'data/sRGB.icc'
@@ -56,9 +61,6 @@ pdfa_def_template = u"""%!
 /ICCProfile $icc_profile
 def

-[$docinfo
-  /DOCINFO pdfmark
-
 % Define an ICC profile :

 [/_objdef {icc_PDFA} /type /stream /OBJ pdfmark
@@ -89,6 +91,7 @@ def
 """


+@deprecated
 def encode_text_string(s: str) -> str:
    """
    Encode text string to hex string for use in a PDF
@@ -134,93 +137,17 @@ def _encode_ascii(s: str) -> str:
    return s.translate(trans).encode('ascii', errors='replace').decode()


-def encode_pdf_date(d: datetime) -> str:
-    """
-    Encode Python datetime object as PDF date string
-
-    From Adobe pdfmark manual:
-    (D:YYYYMMDDHHmmSSOHH'mm')
-    D: is an optional prefix. YYYY is the year. All fields after the year are
-    optional. MM is the month (01-12), DD is the day (01-31), HH is the
-    hour (00-23), mm are the minutes (00-59), and SS are the seconds
-    (00-59). The remainder of the string defines the relation of local
-    time to GMT. O is either + for a positive difference (local time is
-    later than GMT) or - (minus) for a negative difference. HH' is the
-    absolute value of the offset from GMT in hours, and mm' is the
-    absolute value of the offset in minutes. If no GMT information is
-    specified, the relation between the specified time and GMT is
-    considered unknown. Regardless of whether or not GMT
-    information is specified, the remainder of the string should specify
-    the local time.
-    """
-
-    pdfmark_date_fmt = r'%Y%m%d%H%M%S'
-    s = d.strftime(pdfmark_date_fmt)
-
-    tz = d.strftime('%z')
-    if tz == 'Z' or tz == '':
-        # Ghostscript <= 9.23 handles missing timezones incorrectly, so if
-        # timezone is missing, move it into GMT.
-        # https://bugs.ghostscript.com/show_bug.cgi?id=699182
-        s += "+00'00'"
-    else:
-        sign, tz_hours, tz_mins = tz[0], tz[1:3], tz[3:5]
-        s += "{}{}'{}'".format(sign, tz_hours, tz_mins)
-    return s
+@deprecated
+def encode_pdf_date(*args, **kwargs):
+    return _encode_date(*args, **kwargs)


-def decode_pdf_date(s: str) -> datetime:
-    """
-    Decode a pdfmark date to a Python datetime object
-
-    A pdfmark date is a string in a paritcular format. See the pdfmark
-    Reference for the specification.
-
-    """
-    if s.startswith('D:'):
-        s = s[2:]
-
-    # Literal Z00'00', is incorrect but found in the wild,
-    # probably made by OS X Quartz -- standardize
-    if s.endswith("Z00'00'"):
-        s = s.replace("Z00'00'", '+0000')
-    elif s.endswith('Z'):
-        s = s.replace('Z', '+0000')
-
-    s = s.replace("'", "")  # Remove apos from PDF time strings
-
-    return datetime.strptime(s, r'%Y%m%d%H%M%S%z')
+@deprecated
+def decode_pdf_date(*args, **kwargs):
+    return _decode_date(*args, **kwargs)


-def _get_pdfmark_dates(pdfmark):
-    """
-    Encode dates in the expected format for pdfmark Postscript
-
-    The best way to deal with amissing date entry is set it to null, because if
-    the key is omitted Ghostscript will set it to now - we do not want to erase
-    the fact that the value was unknown.  Setting to an empty string breaks
-    Ghostscript 9.22 as reported here:
-    https://bugs.ghostscript.com/show_bug.cgi?id=699182
-    """
-
-    for key in ('/CreationDate', '/ModDate'):
-        if key not in pdfmark:
-            continue
-        if pdfmark[key].strip() == '':
-            yield '  {} null'.format(key)
-            continue
-        date_str = pdfmark[key]
-        if date_str.startswith('D:'):
-            date_str = date_str[2:]
-        try:
-            yield '  {} (D:{})'.format(
-                    key,
-                    encode_pdf_date(decode_pdf_date(date_str)))
-        except ValueError:
-            yield '  {} null'.format(key)
-
-
-def _get_pdfa_def(icc_profile, icc_identifier, pdfmark, ascii_docinfo=False):
+def _get_pdfa_def(icc_profile, icc_identifier, pdfmark=None, ascii_docinfo=None):
    """
    Create a Postscript pdfmark file for Ghostscript.

@@ -230,43 +157,18 @@ def _get_pdfa_def(icc_profile, icc_identifier, pdfmark, ascii_docinfo=False):
    :param icc_profile: filename of the ICC profile to include in pdfmark
    :param icc_identifier: ICC identifier such as 'sRGB'
    :param pdfmark: a dictionary containing keys to include the pdfmark
-    :param ascii_docinfo: if True, the docinfo block must be encoded in pure
-        ASCII and may not contain UTF-16BE-BOM-hex encoded strings, as
-        required for Ghostscript 9.24+
+    :param ascii_docinfo: parameter is no longer meaningful

    :returns: a string containing the entire pdfmark
-
    """

-    # Ghostscript <= 9.21 has a bug where null entries in DOCINFO might produce
-    # ERROR: VMerror (-25) on closing pdfwrite device.
-    # https://bugs.ghostscript.com/show_bug.cgi?id=697684
-    # Work around this by only adding keys that have a nontrivial value
-    docinfo_keys = ('/Title', '/Author', '/Subject', '/Creator', '/Keywords')
-
-    def docinfo_gen():
-        if not ascii_docinfo:
-            docinfo_line_template = '  {key} <{value}>'
-            encode = encode_text_string
-        else:
-            docinfo_line_template = '  {key} ({value})'
-            encode = _encode_ascii
-        yield from _get_pdfmark_dates(pdfmark)
-        for key in docinfo_keys:
-            if key in pdfmark and pdfmark[key].strip() != '':
-                line = docinfo_line_template.format(
-                    key=key, value=encode(pdfmark[key]))
-                yield line
-    docinfo = '\n'.join(docinfo_gen())
-
    t = Template(pdfa_def_template)
    result = t.substitute(icc_profile=icc_profile,
-                          icc_identifier=icc_identifier,
-                          docinfo=docinfo)
+                          icc_identifier=icc_identifier)
    return result


-def generate_pdfa_ps(target_filename, pdfmark, icc='sRGB', ascii_docinfo=False):
+def generate_pdfa_ps(target_filename, pdfmark=None, icc='sRGB', ascii_docinfo=None):
    if icc == 'sRGB':
        icc_profile = SRGB_ICC_PROFILE
    else:
@@ -283,7 +185,7 @@ def generate_pdfa_ps(target_filename, pdfmark, icc='sRGB', ascii_docinfo=False):
    hex_icc_profile = hexlify(bytes_icc_profile)
    icc_profile = '<' + hex_icc_profile.decode('ascii') + '>'

-    ps = _get_pdfa_def(icc_profile, icc, pdfmark, ascii_docinfo=ascii_docinfo)
+    ps = _get_pdfa_def(icc_profile, icc, pdfmark)

    # We should have encoded everything to pure ASCII by this point, and
    # to be safe, only allow ASCII in PostScript
@@ -300,34 +202,16 @@ def file_claims_pdfa(filename):
    This checks if the XMP metadata contains a PDF/A marker.
    """

-    xmp = file_to_dict(filename)
-    if not xmp:
-        return {'pass': False, 'output': 'pdf',
-                'conformance': 'No XMP metadata'}
-
-    if not consts.XMP_NS_PDFA_ID in xmp:
+    pdf = pikepdf.open(filename)
+    pdfmeta = pdf.open_metadata()
+    if not pdfmeta.pdfa_status:
        return {'pass': False, 'output': 'pdf',
                'conformance': 'No PDF/A metadata in XMP'}
-
-    pdfa_node = xmp[consts.XMP_NS_PDFA_ID]
-    def read_node(node, key):
-        return next(
-            (v for k, v, meta in node if k == key), ''
-        )
-
-    part = read_node(pdfa_node, 'pdfaid:part')
-    conformance = read_node(pdfa_node, 'pdfaid:conformance')
-
-    part_conformance = part + conformance
    valid_part_conforms = {'1A', '1B', '2A', '2B', '2U', '3A', '3B', '3U'}
-
-    conformance = 'PDF/A-{}'.format(
-        part_conformance)
-
+    conformance = 'PDF/A-{}'.format(pdfmeta.pdfa_status)
    pdfa_dict = {}
-    if part_conformance in valid_part_conforms:
+    if pdfmeta.pdfa_status in valid_part_conforms:
        pdfa_dict['pass'] = True
        pdfa_dict['output'] = 'pdfa'
    pdfa_dict['conformance'] = conformance
-
    return pdfa_dict
@@ -22,6 +22,7 @@ from enum import Enum
 from math import hypot, isclose
 from pathlib import Path
 from unittest.mock import Mock
+from warnings import warn
 import re

 from pikepdf import PdfMatrix
@@ -30,6 +31,7 @@ import pikepdf
 from . import ghosttext
 from .layout import get_page_analysis, get_text_boxes

+from ..exceptions import EncryptedPdfError
 from ..helpers import fspath


@@ -135,6 +137,13 @@ def _interpret_contents(contentstream, initial_shorthand=UNIT_SQUARE):
    page.

    PDF units suit our needs so we initialize ctm to the identity matrix.
+
+    According to the PDF specification, the maximum stack depth is 32. Other
+    viewers tolerate some amount beyond this.  We issue a warning if the
+    stack depth exceeds the spec limit and set a hard limit beyond this to
+    bound our memory requirements.  If the stack underflows behavior is
+    undefined in the spec, but we just pretend nothing happened and leave the
+    CTM unchanged.
    """

    stack = []
@@ -149,18 +158,20 @@ def _interpret_contents(contentstream, initial_shorthand=UNIT_SQUARE):
    for n, graphobj in enumerate(_normalize_stack(
            pikepdf.parse_content_stream(contentstream, operator_whitelist))):
        operands, operator = graphobj
-
        if operator == 'q':
            stack.append(ctm)
-            if len(stack) > 32:
-                raise RuntimeError(
-                    "PDF graphics stack overflow, operator %i" % n)
+            if len(stack) > 32:  # See docstring
+                if len(stack) > 128:
+                    raise RuntimeError(
+                        "PDF graphics stack overflowed hard limit, operator %i" % n)
+                warn("PDF graphics stack overflowed spec limit")
        elif operator == 'Q':
            try:
                ctm = stack.pop()
            except IndexError:
-                raise RuntimeError(
-                    "PDF graphics stack underflow, operator %i" % n)
+                # Keeping the ctm the same seems to be the only sensible thing
+                # to do. Just pretend nothing happened, keep calm and carry on.
+                warn("PDF graphics stack underflowed - PDF may be malformed")
        elif operator == 'cm':
            ctm = PdfMatrix(operands) @ ctm
        elif operator == 'Do':
@@ -539,12 +550,12 @@ def _pdf_get_pageinfo(pdf, pageno: int, infile, xmltext):
    width_pt = mediabox[2] - mediabox[0]
    height_pt = mediabox[3] - mediabox[1]

-    if xmltext:
+    if xmltext is not None:
        bboxes = ghosttext.page_get_textblocks(
            fspath(infile), pageno, xmltext=xmltext, height=height_pt)
        pageinfo['bboxes'] = bboxes
    else:
-        pscript5_mode = str(pdf.metadata.get('/Creator')).startswith('PScript5')
+        pscript5_mode = str(pdf.docinfo.get('/Creator')).startswith('PScript5')
        miner = get_page_analysis(infile, pageno, pscript5_mode)
        pageinfo['textboxes'] = list(simplify_textboxes(miner))
        bboxes = (box.bbox for box in pageinfo['textboxes'])
@@ -593,6 +604,8 @@ def _pdf_get_all_pageinfo(infile, detailed_analysis=False, log=None):
        log = Mock()

    pdf = pikepdf.open(infile)
+    if pdf.is_encrypted:
+        raise EncryptedPdfError()  # Triggered by encryption with empty passwd
    if detailed_analysis:
        pages_xml = None
    else:
@@ -42,7 +42,7 @@ STRIP_NAME = re.compile(r'[0-9]+')
 #

 def name2unicode(name):
-    """Fix pdfminer's regex in name2unicode function
+    """Fix pdfminer's name2unicode function

    Font cids that are mapped to names of the form /g123 seem to be, by convention
    characters with no corresponding Unicode entry. These can be subsetted fonts
@@ -25,11 +25,13 @@ from unittest.mock import patch
 import datetime

 import pikepdf
+from pikepdf.models.metadata import decode_pdf_date

 from ocrmypdf.exceptions import ExitCode
 from ocrmypdf.helpers import fspath
 from ocrmypdf.pdfa import (
-    file_claims_pdfa, encode_pdf_date, decode_pdf_date, generate_pdfa_ps,
+    file_claims_pdfa,
+    generate_pdfa_ps,
    SRGB_ICC_PROFILE
 )
 from ocrmypdf.exec import ghostscript
@@ -43,6 +45,8 @@ except ImportError:
 # pylint: disable=no-member
 # pylint: disable=w0612

+pytestmark = pytest.mark.filterwarnings('ignore:.*XMLParser.*:DeprecationWarning')
+
 check_ocrmypdf = pytest.helpers.check_ocrmypdf
 run_ocrmypdf = pytest.helpers.run_ocrmypdf
 spoof = pytest.helpers.spoof
@@ -63,7 +67,7 @@ def test_preserve_metadata(spoof_tesseract_noop, output_type,
    pdf_after = pikepdf.open(output)

    for key in ('/Title', '/Author'):
-        assert pdf_before.metadata[key] == pdf_after.metadata[key]
+        assert pdf_before.docinfo[key] == pdf_after.docinfo[key]

    pdfa_info = file_claims_pdfa(str(output))
    assert pdfa_info['output'] == output_type
@@ -90,15 +94,12 @@ def test_override_metadata(spoof_tesseract_noop, output_type, resources,
    before = pikepdf.open(input_file)
    after = pikepdf.open(outpdf)

-    if ghostscript.version() >= '9.24':
-        pytest.xfail('Ghostscript 9.24+ does not support Unicode DOCINFO')
+    assert after.docinfo.Title == german, after.docinfo
+    assert after.docinfo.Author == chinese, after.docinfo
+    assert after.docinfo.get('/Keywords', '') == ''

-    assert after.metadata.Title == german, after.metadata
-    assert after.metadata.Author == chinese, after.metadata
-    assert after.metadata.get('/Keywords', '') == ''
-
-    before_date = decode_pdf_date(str(before.metadata.CreationDate))
-    after_date = decode_pdf_date(str(after.metadata.CreationDate))
+    before_date = decode_pdf_date(str(before.docinfo.CreationDate))
+    after_date = decode_pdf_date(str(after.docinfo.CreationDate))
    assert before_date == after_date

    pdfa_info = file_claims_pdfa(outpdf)
@@ -162,11 +163,7 @@ def test_creation_date_preserved(spoof_tesseract_noop, output_type, resources,
    after = pdf_after.trailer.get('/Info', {})

    if not before:
-        # If there was input creation date, none should be output
-        # because of Ghostscript quirks we set it to null
-        # This test would be better if we had a test file with /DocumentInfo but
-        # no /CreationDate, which we don't
-        assert after.get('/CreationDate', '') == ''
+        assert after.get('/CreationDate', '') != ''
    else:
        # We expect that the creation date stayed the same
        date_before = decode_pdf_date(str(before['/CreationDate']))
@@ -27,7 +27,6 @@ import shutil
 import pytest
 import img2pdf
 import sys
-import PyPDF2 as pypdf
 import pikepdf
 import pickle

@@ -196,3 +195,22 @@ def test_corrupt_font_detection(resources, testfile):

    pdf = pdfinfo.PdfInfo(filename, detailed_page_analysis=True)
    assert pdf[0].has_corrupt_text
+
+
+def test_stack_abuse():
+    p = pikepdf.Pdf.new()
+
+    stream = pikepdf.Stream(p, b'q ' * 35)
+    with pytest.warns(None) as record:
+        pdfinfo._interpret_contents(stream)
+    assert 'overflowed' in str(record[0].message)
+
+    stream = pikepdf.Stream(p, b'q Q Q Q Q')
+    with pytest.warns(None) as record:
+        pdfinfo._interpret_contents(stream)
+    assert 'underflowed' in str(record[0].message)
+
+    stream = pikepdf.Stream(p, b'q ' * 135)
+    with pytest.warns(None):
+        with pytest.raises(RuntimeError):
+            pdfinfo._interpret_contents(stream)
Author	SHA1	Message	Date
James R. Barlow	ab632f57cd	v7.4.0 release notes	2018-12-15 15:27:23 -08:00
James R. Barlow	13d20bd993	pdfinfo: tolerate PDFs that overflow and underflow the graphics stack	2018-12-15 15:10:29 -08:00
James R. Barlow	b973208137	Require pikepdf 0.9.1	2018-12-15 14:23:10 -08:00
James R. Barlow	942abf8074	Fix reqs/main.txt for pikepdf 0.9.0	2018-12-14 23:29:26 -08:00
James R. Barlow	ed9bb985e2	Fix pikepdf 0.9.0	2018-12-14 23:21:13 -08:00
James R. Barlow	5a7a8e573b	Require pikepdf 0.9.0	2018-12-14 23:06:57 -08:00
James R. Barlow	ce878db913	Rename to polyglot.dockerfile	2018-12-14 23:06:29 -08:00
James R. Barlow	a3d58683b2	Update webservice.py with separate license	2018-12-14 23:05:54 -08:00
James R. Barlow	039e8ca7e7	Merge branches 'feature/newer-pike' and 'feature/webapp'	2018-12-14 18:08:31 -08:00
James R. Barlow	0ebbd4e21b	Don't open encrypted files, even if password is empty	2018-12-13 22:48:00 -08:00
James R. Barlow	2cb75f6076	Refactor pipeline to make PDF/A conversion a separate step	2018-12-13 20:48:48 -08:00
James R. Barlow	857d871364	Fix regression on Ghostscript path	2018-12-13 20:36:41 -08:00
James R. Barlow	632dab2cc0	Replace Ghostscript DOCINFO and fix 9.25 metadata date regression We no longer use Ghostscript to manage PDF metadata, instead omitting the DOCINFO segment from the pdfmark file we generate. Instead all of the relevant metadata code has been migrated to pikepdf, and we use that API. This should be more consistent and fixes the Ghostscript version-depedent quirks. Also removes our python-xmp-toolkit dependency, except for testing.	2018-12-13 18:13:30 -08:00
James R. Barlow	7647918f2d	setup: suppress XMLParser() warning - defusedxml related	2018-12-12 22:13:32 -08:00
James R. Barlow	75c5d8055c	pdfinfo: fix FutureWarning	2018-12-12 22:12:14 -08:00
James R. Barlow	a938bbea55	Remove more libxmp dependencies	2018-12-12 22:02:35 -08:00
James R. Barlow	414407fbd6	Deprecate encode/decode_pdf_date and remap to pikepdf version	2018-12-12 22:01:21 -08:00
James R. Barlow	076fc717df	pdfa: replace PDF/A checking with pikepdf implementation	2018-12-12 21:41:16 -08:00
James R. Barlow	2a04b2d82b	Rename webapp to webservice	2018-12-12 21:29:05 -08:00
James R. Barlow	065db414c0	webapp docker: Build from polyglot	2018-12-12 21:24:04 -08:00
James R. Barlow	19a054a78b	Add webapp stuff	2018-12-10 20:03:52 -08:00
James R. Barlow	9df24a81b7	Fix comment in layout.py	2018-11-28 15:16:34 -08:00
James R. Barlow	40c0acd3f2	Support using --force-ocr and --threshold or --mask-barcodes together	2018-11-28 15:16:24 -08:00
James R. Barlow	20db7f0a8f	leptonica: delete file junkpixt.png if created	2018-11-28 13:47:55 -08:00