Update reqs

README.md

@@ -38,7 +38,7 @@ Main features
 - Keeps the exact resolution of the original embedded images
 - When possible, inserts OCR information as a "lossless" operation without disrupting any other content
 - Optimizes PDF images, often producing files smaller than the input file
-- If requested deskews and/or cleans the image before performing OCR
+- If requested, deskews and/or cleans the image before performing OCR
 - Validates input and output files
 - Distributes work across all available CPU cores
 - Uses [Tesseract OCR](https://github.com/tesseract-ocr/tesseract) engine to recognize more than [100 languages](https://github.com/tesseract-ocr/tessdata)
@@ -50,7 +50,7 @@ For details: please consult the [documentation](https://ocrmypdf.readthedocs.io/
 Motivation
 ----------
 
-I searched the web for a free command line tool to OCR PDF files: I found many, but none of them were really satisfying.
+I searched the web for a free command line tool to OCR PDF files: I found many, but none of them were really satisfying:
 
 - Either they produced PDF files with misplaced text under the image (making copy/paste impossible)
 - Or they did not handle accents and multilingual characters
@@ -97,7 +97,10 @@ OCRmyPDF uses Tesseract for OCR, and relies on its language packs. For Linux use
 apt-cache search tesseract-ocr
 
 # Debian/Ubuntu users
-apt-get install tesseract-ocr-chi-sim  # Example: Install Chinese Simplified language back
+apt-get install tesseract-ocr-chi-sim  # Example: Install Chinese Simplified language pack
+
+# Arch Linux users
+pacman -S tesseract-data-eng tesseract-data-deu # Example: Install the English and German language packs
 ```
 
 You can then pass the `-l LANG` argument to OCRmyPDF to give a hint as to what languages it should search for. Multiple languages can be requested.
@@ -105,7 +108,7 @@ You can then pass the `-l LANG` argument to OCRmyPDF to give a hint as to what l
 Documentation and support
 -------------------------
 
-Once ocrmypdf is installed, the built-in help which explains the command syntax and options can be accessed via:
+Once OCRmyPDF is installed, the built-in help which explains the command syntax and options can be accessed via:
 
 ```bash
 ocrmypdf --help
@@ -118,20 +121,20 @@ Please report issues on our [GitHub issues](https://github.com/jbarlow83/OCRmyPD
 Requirements
 ------------
 
-In addition to the required Python version (3.6+), OCRmyPDF requires external program installations of Ghostscript, Tesseract OCR, QPDF, and Leptonica. ocrmypdf is pure Python, but uses CFFI to portably generate library bindings. OCRmyPDF works on pretty much  everything: Linux, macOS, Windows and FreeBSD.
+In addition to the required Python version (3.6+), OCRmyPDF requires external program installations of Ghostscript, Tesseract OCR, QPDF, and Leptonica. OCRmyPDF is pure Python, but uses CFFI to portably generate library bindings. OCRmyPDF works on pretty much everything: Linux, macOS, Windows and FreeBSD.
 
 Press & Media
 -------------
 
 - [Going paperless with OCRmyPDF](https://medium.com/@ikirichenko/going-paperless-with-ocrmypdf-e2f36143f46a)
 - [Converting a scanned document into a compressed searchable PDF with redactions](https://medium.com/@treyharris/converting-a-scanned-document-into-a-compressed-searchable-pdf-with-redactions-63f61c34fe4c)
-- [c't 1-2014, page 59](http://heise.de/-2279695): Detailed presentation of OCRmyPDF v1.0 in the leading German IT magazine c't
-- [heise Open Source, 09/2014: Texterkennung mit OCRmyPDF](http://heise.de/-2356670)
+- [c't 1-2014, page 59](https://heise.de/-2279695): Detailed presentation of OCRmyPDF v1.0 in the leading German IT magazine c't
+- [heise Open Source, 09/2014: Texterkennung mit OCRmyPDF](https://heise.de/-2356670)
 
 Business enquiries
 ------------------
 
-OCRmyPDF would not be the software that it is today is without companies and users choosing to provide support for feature development and consulting enquiries. We are happy to discuss all enquiries, whether for extending the existing feature set, or integrating OCRmyPDF into a larger system.
+OCRmyPDF would not be the software that it is today without companies and users choosing to provide support for feature development and consulting enquiries. We are happy to discuss all enquiries, whether for extending the existing feature set, or integrating OCRmyPDF into a larger system.
 
 License
 -------

docs/batch.rst

@@ -210,6 +210,9 @@ be launched as follows:
         -v <path to files to convert>:/input \
         -v <path to store results>:/output \
         -e OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1 \
+        -e OCR_ON_SUCCESS_DELETE=1 \
+        -e OCR_DESKEW=1 \
+        -e PYTHONUNBUFFERED=1 \
         -it --entrypoint python3 \
         jbarlow83/ocrmypdf \
         watcher.py
@@ -224,6 +227,9 @@ convert it to a OCRed PDF in ``/output/``. The parameters to this image are:
     "``-v <path to files to convert>:/input``", "Files placed in this location will be OCRed"
     "``-v <path to store results>:/output``", "This is where OCRed files will be stored"
     "``-e OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1``", "This will place files in the output in {output}/{year}/{month}/{filename}"
+    "``-e OCR_ON_SUCCESS_DELETE=1``", "This will delete the input file if the exit code is 0 (OK)"
+    "``-e OCR_DESKEW=1``", "This will enable deskew for crooked PDFs"
+    "``-e PYTHONBUFFERED=1``", "This will force STDOUT to be unbuffered and allow you to see messages in docker logs"
 
 This service relies on polling to check for changes to the filesystem. It
 may not be suitable for some environments, such as filesystems shared on a

docs/installation.rst

@@ -141,18 +141,20 @@ first install the system version to get most of the dependencies:
 
 .. code-block:: bash
 
-    sudo apt-get update
-    sudo apt-get install \
-        ocrmypdf
-
-There are a few system dependency changes since ocrmypdf 6.1.2. Let's
-get these, too.
-
-.. code-block:: bash
-
-    sudo apt-get install \
+    sudo apt-get -y update
+    sudo apt-get -y install \
+        ghostscript \
+        icc-profiles-free \
+        liblept5 \
         libxml2 \
-        pngquant
+        pngquant \
+        python3-cffi \
+        python3-distutils \
+        python3-pkg-resources \
+        python3-reportlab \
+        qpdf \
+        tesseract-ocr \
+        zlib1g
 
 We will need a newer version of ``pip`` then was available for Ubuntu 18.04:
 

docs/release_notes.rst

@@ -13,6 +13,24 @@ Note that it is licensed under GPLv3, so scripts that
 ``import ocrmypdf`` and are released publicly should probably also be
 licensed under GPLv3.
 
+v9.6.0
+======
+
+-  Fixed a regression with transferring metadata from the input PDF to the output
+   PDF in certain situations.
+-  pdfminer.six is now supported up to version 2020-01-24.
+-  Messages are explaining page rotation decisions are now shown at the standard
+   verbosity level again when ``--rotate-pages``. In some previous version they
+   were set to debug level messages that only appeared with the parameter ``-v1``.
+-  Improvements to ``misc/watcher.py``. Thanks to @ianalexander and @svenihoney.
+-  Documentation improvements.
+
+v9.5.0
+======
+
+-  Added API functions to measure OCR quality.
+-  Modest improvements to handling PDFs with difficult/non compliant metadata.
+
 v9.4.0
 ======
 

misc/watcher.py

@@ -13,51 +13,114 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+import logging
 import os
 import time
 from datetime import datetime
 from pathlib import Path
 
+import pikepdf
 from watchdog.events import PatternMatchingEventHandler
 from watchdog.observers import Observer
 
 import ocrmypdf
 
+# pylint: disable=logging-format-interpolation
+
 INPUT_DIRECTORY = os.getenv('OCR_INPUT_DIRECTORY', '/input')
 OUTPUT_DIRECTORY = os.getenv('OCR_OUTPUT_DIRECTORY', '/output')
 OUTPUT_DIRECTORY_YEAR_MONTH = bool(os.getenv('OCR_OUTPUT_DIRECTORY_YEAR_MONTH', False))
+ON_SUCCESS_DELETE = bool(os.getenv('OCR_ON_SUCCESS_DELETE', False))
+DESKEW = bool(os.getenv('OCR_DESKEW', False))
+POLL_NEW_FILE_SECONDS = os.getenv('OCR_POLL_NEW_FILE_SECONDS', 1)
+LOGLEVEL = os.environ.get('OCR_LOGLEVEL', 'INFO').upper()
 PATTERNS = ['*.pdf']
 
+log = logging.getLogger('ocrmypdf-watcher')
 
-def execute_ocrmypdf(file_path):
-    filename = Path(file_path).name
+
+def get_output_dir(root, basename):
     if OUTPUT_DIRECTORY_YEAR_MONTH:
         today = datetime.today()
-        output_directory_year_month = Path(
-            f'{OUTPUT_DIRECTORY}/{today.year}/{today.month}'
+        output_directory_year_month = (
+            Path(root) / str(today.year) / f'{today.month:02d}'
         )
         if not output_directory_year_month.exists():
             output_directory_year_month.mkdir(parents=True, exist_ok=True)
-        output_path = Path(output_directory_year_month) / filename
+        output_path = Path(output_directory_year_month) / basename
     else:
-        output_path = Path(OUTPUT_DIRECTORY) / filename
-    print(f'New file: {file_path}.\nAttempting to OCRmyPDF to: {output_path}')
-    ocrmypdf.ocr(file_path, output_path)
+        output_path = Path(OUTPUT_DIRECTORY) / basename
+    return output_path
+
+
+def wait_for_file_ready(file_path):
+    # This loop waits to make sure that the file is completely loaded on
+    # disk before attempting to read. Docker sometimes will publish the
+    # watchdog event before the file is actually fully on disk, causing
+    # pikepdf to fail.
+
+    retries = 5
+    while retries:
+        try:
+            pdf = pikepdf.open(file_path)
+        except (FileNotFoundError, pikepdf.PdfError) as e:
+            log.info(f"File {file_path} is not ready yet")
+            log.debug("Exception was", exc_info=e)
+            time.sleep(POLL_NEW_FILE_SECONDS)
+            retries -= 1
+        else:
+            pdf.close()
+            return True
+
+    return False
+
+
+def execute_ocrmypdf(file_path):
+    file_path = Path(file_path)
+    output_path = get_output_dir(OUTPUT_DIRECTORY, file_path.name)
+
+    log.info("-" * 20)
+    log.info(f'New file: {file_path}. Waiting until fully loaded...')
+    if not wait_for_file_ready(file_path):
+        log.info(f"Gave up waiting for {file_path} to become ready")
+        return
+    log.info(f'Attempting to OCRmyPDF to: {output_path}')
+    exit_code = ocrmypdf.ocr(
+        input_file=file_path, output_file=output_path, deskew=DESKEW
+    )
+    if exit_code == 0 and ON_SUCCESS_DELETE:
+        log.info(f'OCR is done. Deleting: {file_path}')
+        file_path.unlink()
+    else:
+        log.info('OCR is done')
 
 
 class HandleObserverEvent(PatternMatchingEventHandler):
     def on_any_event(self, event):
-        if event.event_type in ['created', 'modified']:
+        if event.event_type in ['created']:
             execute_ocrmypdf(event.src_path)
 
 
-if __name__ == "__main__":
-    print(
+def main():
+    ocrmypdf.configure_logging(
+        verbosity=ocrmypdf.Verbosity.default, manage_root_logger=True
+    )
+    log.info(
         f"Starting OCRmyPDF watcher with config:\n"
         f"Input Directory: {INPUT_DIRECTORY}\n"
         f"Output Directory: {OUTPUT_DIRECTORY}\n"
         f"Output Directory Year & Month: {OUTPUT_DIRECTORY_YEAR_MONTH}"
     )
+    log.debug(
+        f"INPUT_DIRECTORY: {INPUT_DIRECTORY}\n"
+        f"OUTPUT_DIRECTORY: {OUTPUT_DIRECTORY}\n"
+        f"OUTPUT_DIRECTORY_YEAR_MONTH: {OUTPUT_DIRECTORY_YEAR_MONTH}\n"
+        f"ON_SUCCESS_DELETE: {ON_SUCCESS_DELETE}\n"
+        f"DESKEW: {DESKEW}\n"
+        f"POLL_NEW_FILE_SECONDS: {POLL_NEW_FILE_SECONDS}\n"
+        f"LOGLEVEL: {LOGLEVEL}\n"
+    )
+
     handler = HandleObserverEvent(patterns=PATTERNS)
     observer = Observer()
     observer.schedule(handler, INPUT_DIRECTORY, recursive=True)
@@ -68,3 +131,7 @@ if __name__ == "__main__":
     except KeyboardInterrupt:
         observer.stop()
     observer.join()
+
+
+if __name__ == "__main__":
+    main()

requirements/main.txt

@@ -1,10 +1,10 @@
 # requirements.txt can be used to replicate the developer's build environment
 # setup.py lists a separate set of requirements that are looser to simplify
 # installation
-cffi == 1.13.2
+cffi == 1.14.0
 img2pdf == 0.3.3
-pdfminer.six == 20200104
-pikepdf == 1.8.2
+pdfminer.six == 20200124
+pikepdf == 1.10.1
 Pillow == 7.0.0
-reportlab == 3.5.32
-tqdm == 4.41.1
+reportlab == 3.5.34
+tqdm == 4.42.1

setup.py

@@ -98,7 +98,7 @@ setup(
         'chardet >= 3.0.4, < 4',  # unlisted requirement of pdfminer.six 20181108
         'cffi >= 1.9.1',  # must be a setup and install requirement
         'img2pdf >= 0.3.0, < 0.4',  # pure Python, so track HEAD closely
-        'pdfminer.six >= 20181108, <= 20200104',
+        'pdfminer.six >= 20181108, <= 20200124',
         'pikepdf >= 1.8.1, < 2',
         'Pillow >= 6.2.0',
         'reportlab >= 3.3.0',  # oldest released version with sane image handling

src/ocrmypdf/__init__.py

@@ -23,6 +23,7 @@ from .exceptions import (
     DpiError,
     EncryptedPdfError,
     ExitCode,
+    ExitCodeException,
     InputFileError,
     MissingDependencyError,
     OutputFileAccessError,

src/ocrmypdf/_pipeline.py

@@ -329,6 +329,35 @@ def rasterize_preview(input_file, page_context):
     return output_file
 
 
+def describe_rotation(page_context, orient_conf, correction):
+    """
+    Describe the page rotation we are going to perform.
+    """
+    direction = {0: '⇧', 90: '⇨', 180: '⇩', 270: '⇦'}
+    turns = {0: ' ', 90: '⬏', 180: '↻', 270: '⬑'}
+
+    existing_rotation = page_context.pageinfo.rotation
+    action = ''
+    if orient_conf.confidence >= page_context.options.rotate_pages_threshold:
+        if correction != 0:
+            action = 'will rotate ' + turns[correction]
+        else:
+            action = 'rotation appears correct'
+    else:
+        if correction != 0:
+            action = 'confidence too low to rotate'
+        else:
+            action = 'no change'
+
+    facing = ''
+
+    if existing_rotation != 0:
+        facing = f"with existing rotation {direction.get(existing_rotation, '?')}, "
+    facing += f"page is facing {direction.get(orient_conf.angle, '?')}"
+
+    return f"{facing}, confidence {orient_conf.confidence:.2f} - {action}"
+
+
 def get_orientation_correction(preview, page_context):
     """
     Work out orientation correct for each page.
@@ -355,44 +384,14 @@ def get_orientation_correction(preview, page_context):
         tesseract_env=page_context.options.tesseract_env,
     )
 
-    direction = {0: '⇧', 90: '⇨', 180: '⇩', 270: '⇦'}
-
-    existing_rotation = page_context.pageinfo.rotation
-
     correction = orient_conf.angle % 360
-
-    apply_correction = False
-    action = ''
-    if orient_conf.confidence >= page_context.options.rotate_pages_threshold:
-        if correction != 0:
-            apply_correction = True
-            action = ' - will rotate'
-        else:
-            action = ' - rotation appears correct'
-    else:
-        if correction != 0:
-            action = ' - confidence too low to rotate'
-        else:
-            action = ' - no change'
-
-    facing = ''
-    if existing_rotation != 0:
-        facing = 'with existing rotation {}, '.format(
-            direction.get(existing_rotation, '?')
-        )
-    facing += 'page is facing {}'.format(direction.get(orient_conf.angle, '?'))
-
-    page_context.log.debug(
-        '{pagenum:4d}: {facing}, confidence {conf:.2f}{action}'.format(
-            pagenum=page_context.pageinfo.pageno,
-            facing=facing,
-            conf=orient_conf.confidence,
-            action=action,
-        )
-    )
-
-    if apply_correction:
+    page_context.log.info(describe_rotation(page_context, orient_conf, correction))
+    if (
+        orient_conf.confidence >= page_context.options.rotate_pages_threshold
+        and correction != 0
+    ):
         return correction
+
     return 0
 
 
@@ -727,47 +726,53 @@ def should_linearize(working_file, context):
 def metadata_fixup(working_file, context):
     output_file = context.get_path('metafix.pdf')
     options = context.options
-    original = pikepdf.open(context.origin)
-    docinfo = get_docinfo(original, options)
-    pdf = pikepdf.open(working_file)
-    with pdf.open_metadata() as meta:
-        meta.load_from_docinfo(docinfo, delete_missing=False)
-        # If xmp:CreateDate is missing, set it to the modify date to
-        # match Ghostscript, for consistency
-        if 'xmp:CreateDate' not in meta:
-            meta['xmp:CreateDate'] = meta.get('xmp:ModifyDate', '')
 
-        meta_original = original.open_metadata()
-        not_copied = set(meta_original.keys()) - set(meta.keys())
-        if not_copied:
-            if options.output_type.startswith('pdfa'):
-                context.log.warning(
-                    "Some input metadata could not be copied because it is not "
-                    "permitted in PDF/A. You may wish to examine the output "
-                    "PDF's XMP metadata."
-                )
-                context.log.debug(
-                    "The following metadata fields were not copied: %r", not_copied
-                )
-            else:
-                context.log.error(
-                    "Some input metadata could not be copied."
-                    "You may wish to examine the output PDF's XMP metadata."
-                )
-                context.log.info(
-                    "The following metadata fields were not copied: %r", not_copied
-                )
-    pdf.save(
-        output_file,
-        compress_streams=True,
-        preserve_pdfa=True,
-        object_stream_mode=pikepdf.ObjectStreamMode.generate,
-        linearize=(  # Don't linearize if optimize() will be linearizing too
-            should_linearize(working_file, context) if options.optimize == 0 else False
-        ),
-    )
-    original.close()
-    pdf.close()
+    def report_on_metadata(missing):
+        if not missing:
+            return
+        if options.output_type.startswith('pdfa'):
+            context.log.warning(
+                "Some input metadata could not be copied because it is not "
+                "permitted in PDF/A. You may wish to examine the output "
+                "PDF's XMP metadata."
+            )
+            context.log.debug(
+                "The following metadata fields were not copied: %r", missing
+            )
+        else:
+            context.log.error(
+                "Some input metadata could not be copied."
+                "You may wish to examine the output PDF's XMP metadata."
+            )
+            context.log.info(
+                "The following metadata fields were not copied: %r", missing
+            )
+
+    with pikepdf.open(context.origin) as original, pikepdf.open(working_file) as pdf:
+        docinfo = get_docinfo(original, options)
+        with pdf.open_metadata() as meta:
+            meta.load_from_docinfo(docinfo, delete_missing=False, raise_failure=False)
+            # If xmp:CreateDate is missing, set it to the modify date to
+            # match Ghostscript, for consistency
+            if 'xmp:CreateDate' not in meta:
+                meta['xmp:CreateDate'] = meta.get('xmp:ModifyDate', '')
+
+            meta_original = original.open_metadata()
+            missing = set(meta_original.keys()) - set(meta.keys())
+            report_on_metadata(missing)
+
+        pdf.save(
+            output_file,
+            compress_streams=True,
+            preserve_pdfa=True,
+            object_stream_mode=pikepdf.ObjectStreamMode.generate,
+            linearize=(  # Don't linearize if optimize() will be linearizing too
+                should_linearize(working_file, context)
+                if options.optimize == 0
+                else False
+            ),
+        )
+
     return output_file
 
 

src/ocrmypdf/_sync.py

@@ -231,7 +231,7 @@ def exec_concurrent(context):
     # Run exec_page_sync on every page context
     max_workers = min(len(context.pdfinfo), context.options.jobs)
     if max_workers > 1:
-        context.log.info("Start processing %d pages concurrent", max_workers)
+        context.log.info("Start processing %d pages concurrently", max_workers)
 
     # Tesseract 4.x can be multithreaded, and we also run multiple workers. We want
     # to manage how many threads it uses to avoid creating total threads than cores.
@@ -381,6 +381,7 @@ def run_pipeline(options, api=False):
             detailed_page_analysis=options.redo_ocr,
             progbar=options.progress_bar,
         )
+
         context = PDFContext(options, work_folder, origin_pdf, pdfinfo)
 
         # Validate options are okay for this pdf

src/ocrmypdf/api.py

@@ -18,11 +18,10 @@
 import logging
 import os
 import sys
-import warnings
 from contextlib import suppress
 from enum import IntEnum
 from pathlib import Path
-from typing import Dict, List, Optional
+from typing import Dict, List
 
 from tqdm import tqdm
 

src/ocrmypdf/quality.py

@@ -0,0 +1,60 @@
+# © 2020 James R. Barlow: github.com/jbarlow83
+#
+# This file is part of OCRmyPDF.
+#
+# OCRmyPDF is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# OCRmyPDF is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+
+import re
+from typing import Iterable
+
+"""Utilities to measure OCR quality"""
+
+
+class OcrQualityDictionary:
+    """Manages a dictionary for simple OCR quality checks."""
+
+    def __init__(self, *, wordlist: Iterable[str] = []):
+        """Construct a dictionary from a list of words.
+
+        Words for which capitalization is important should be capitalized in the
+        dictionary. Words that contain spaces or other punctuation will never match.
+        """
+        self.dictionary = set()
+        self.dictionary.update(w for w in wordlist)
+
+    def measure_words_matched(self, ocr_text: str) -> float:
+        """Check how many unique words in the OCR text match a dictionary.
+
+        Words with mixed capitalized are only considered a match if the test word
+        matches that capitalization.
+
+        Returns:
+            number of words that match / number
+        """
+        text = re.sub(r"[0-9_]+", ' ', ocr_text)
+        text = re.sub(r'\W+', ' ', text)
+        text_words_list = re.split(r'\s+', text)
+        text_words = {w for w in text_words_list if len(w) >= 3}
+
+        matches = 0
+        for w in text_words:
+            if w in self.dictionary or (
+                w != w.lower() and w.lower() in self.dictionary
+            ):
+                matches += 1
+        if matches > 0:
+            hit_ratio = matches / len(text_words)
+        else:
+            hit_ratio = 0.0
+        return hit_ratio

tests/test_quality.py

@@ -0,0 +1,35 @@
+# © 2020 James R. Barlow: github.com/jbarlow83
+#
+# This file is part of OCRmyPDF.
+#
+# OCRmyPDF is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# OCRmyPDF is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+
+import pytest
+
+import ocrmypdf.quality as qual
+
+
+def test_quality_measurement():
+    oqd = qual.OcrQualityDictionary(
+        wordlist=["words", "words", "quick", "brown", "fox", "dog", "lazy"]
+    )
+    assert len(oqd.dictionary) == 6  # 6 unique
+
+    assert (
+        oqd.measure_words_matched("The quick brown fox jumps quickly over the lazy dog")
+        == 0.5
+    )
+    assert oqd.measure_words_matched("12345 10% _f  7fox -brown   | words") == 1.0
+
+    assert oqd.measure_words_matched("quick quick quick") == 1.0