v12.0.3 release notes

readme: mention LinuxUser article

README.md

@@ -115,6 +115,7 @@ In addition to the required Python version (3.6+), OCRmyPDF requires external pr
 - [heise Open Source, 09/2014: Texterkennung mit OCRmyPDF](https://heise.de/-2356670)
 - [heise Durchsuchbare PDF-Dokumente mit OCRmyPDF erstellen](https://www.heise.de/ratgeber/Durchsuchbare-PDF-Dokumente-mit-OCRmyPDF-erstellen-4607592.html)
 - [Excellent Utilities: OCRmyPDF](https://www.linuxlinks.com/excellent-utilities-ocrmypdf-add-ocr-text-layer-scanned-pdfs/)
+- [LinuxUser Texterkennung mit OCRmyPDF und Scanbd automatisieren](https://www.linux-community.de/ausgaben/linuxuser/2021/06/texterkennung-mit-ocrmypdf-und-scanbd-automatisieren/)  
 
 ## Business enquiries
 

docs/advanced.rst

@@ -228,8 +228,8 @@ preprocessing is specified, then the image layer is a new PDF.
 Unlike ``sandwich`` this renderer is implemented within OCRmyPDF; anyone
 looking to customize how OCR is presented should look here. A major
 disadvantage of this renderer is it not capable of correctly handling
-text outside the Latin alphabet. Pull requests to improve the situation
+text outside the Latin alphabet (specifically, it supports the ISO 8859-1
-are welcome.
+character). Pull requests to improve the situation are welcome.
 
 Currently, this renderer has the best compatibility with Mozilla's
 PDF.js viewer.

docs/introduction.rst

@@ -139,7 +139,7 @@ Limitations
 OCRmyPDF is limited by the Tesseract OCR engine. As such it experiences
 these limitations, as do any other programs that rely on Tesseract:
 
--  The OCR is not as accurate as commercial solutions such as Abbyy.
+-  The OCR is not as accurate as commercial OCR solutions.
 -  It is not capable of recognizing handwriting.
 -  It may find gibberish and report this as OCR output.
 -  If a document contains languages outside of those given in the

docs/pdfsecurity.rst

@@ -128,8 +128,9 @@ Commercial alternatives
 The author also provides professional services that include OCR and
 building databases around PDFs, and is happy to provide consultation.
 
-Abbyy Cloud OCR is a viable commercial alternative with a web services
+Abbyy Cloud OCR is viable commercial alternative with a web services
-API.
+API. Amazon Textract, Google Cloud Vision, and Microsoft Azure 
+Computer Vision provide advanced OCR but have less PDF rendering capability.
 
 Password protection, digital signatures and certification
 =========================================================

docs/release_notes.rst

@@ -12,6 +12,14 @@ may be unreliable. Use the API to depend on precise behavior.
 The public API may be useful in scripts that launch OCRmyPDF processes or that
 wish to use some of its features for working with PDFs.
 
+v12.0.3
+=======
+
+-  Expand the list of languages supported by the hocr PDF renderer.
+   Several languages were previously considered not supported, particularly those
+   non-European languages that use the Latin alphabet.
+-  Fixed a case where the exception stack trace was suppressed in verbose mode.
+-  Improved documentation around commercial OCR.
 
 v12.0.2
 =======

src/ocrmypdf/_sync.py

@@ -406,7 +406,9 @@ def run_pipeline(options, *, plugin_manager, api=False):
             log.error("KeyboardInterrupt")
         return ExitCode.ctrl_c
     except (ExitCodeException if not api else NeverRaise) as e:
-        if str(e):
+        if options.verbose >= 1:
+            log.exception("ExitCodeException")
+        elif str(e):
             log.error("%s: %s", type(e).__name__, str(e))
         else:
             log.error(type(e).__name__)

src/ocrmypdf/_validation.py

@@ -32,12 +32,12 @@ from ocrmypdf.helpers import (
     monotonic,
     safe_symlink,
 )
+from ocrmypdf.hocrtransform import HOCR_OK_LANGS
 from ocrmypdf.subprocess import check_external_program
 
 # -------------
 # External dependencies
 
-HOCR_OK_LANGS = frozenset(['eng', 'deu', 'spa', 'ita', 'por'])
 DEFAULT_LANGUAGE = 'eng'  # Enforce English hegemony
 
 log = logging.getLogger(__name__)
@@ -279,7 +279,7 @@ def check_closed_streams(options):  # pragma: no cover
     Attempting to a fork/exec a new Python process when any of std{in,out,err}
     are closed or not flushable for some reason may raise an exception.
     Fix this by opening devnull if the handle seems to be closed.  Do this
-    globally to avoid tracking places all places that fork.
+    globally to avoid tracking all places that fork.
 
     Seems to be specific to multiprocessing.Process not all Python process
     forkers.

src/ocrmypdf/hocrtransform.py

@@ -41,6 +41,62 @@ from reportlab.lib.colors import black, cyan, magenta, red
 from reportlab.lib.units import inch
 from reportlab.pdfgen.canvas import Canvas
 
+# According to Wikipedia these languages are supported in the ISO-8859-1 character
+# set, meaning reportlab can generate them and they are compatible with hocr,
+# assuming Tesseract has the necessary languages installed. Note that there may
+# not be language packs for them.
+HOCR_OK_LANGS = frozenset(
+    [
+        # Languages fully covered by Latin-1:
+        'afr',  # Afrikaans
+        'alb',  # Albanian
+        'ast',  # Leonese
+        'baq',  # Basque
+        'bre',  # Breton
+        'cos',  # Corsican
+        'eng',  # English
+        'eus',  # Basque
+        'fao',  # Faoese
+        'gla',  # Scottish Gaelic
+        'glg',  # Galician
+        'glv',  # Manx
+        'ice',  # Icelandic
+        'ind',  # Indonesian
+        'isl',  # Icelandic
+        'ita',  # Italian
+        'ltz',  # Luxembourgish
+        'mal',  # Malay Rumi
+        'mga',  # Irish
+        'nor',  # Norwegian
+        'oci',  # Occitan
+        'por',  # Portugeuse
+        'roh',  # Romansh
+        'sco',  # Scots
+        'sma',  # Sami
+        'spa',  # Spanish
+        'sqi',  # Albanian
+        'swa',  # Swahili
+        'swe',  # Swedish
+        'tgl',  # Tagalog
+        'wln',  # Walloon
+        # Languages supported by Latin-1 except for a few rare characters that OCR
+        # is probably not trained to recognize anyway:
+        'cat',  # Catalan
+        'cym',  # Welsh
+        'dan',  # Danish
+        'deu',  # German
+        'dut',  # Dutch
+        'est',  # Estonian
+        'fin',  # Finnish
+        'fra',  # French
+        'hun',  # Hungarian
+        'kur',  # Kurdish
+        'nld',  # Dutch
+        'wel',  # Welsh
+    ]
+)
+
+
 Element = ElementTree.Element