fix(joex): migrate office conversion defaults to unoserver

Changelog.md

@@ -20,6 +20,13 @@
 
 - Improve performance of custom field statistics on large databases by
   scoping `custom_field_value` lookups to matching items only.
+- Migrate office conversion from deprecated `unoconv` to
+  [unoserver](https://github.com/unoconv/unoserver/) / `unoconvert`
+  (#3345, #3293). The HOCON key `docspell.joex.convert.unoconv` is
+  kept for compatibility; the default program/args now call
+  `unoconvert`. **Breaking:** joex must run a `unoserver` daemon
+  (official Docker images start it in the entrypoint). Custom images
+  that relied on `unoconv -l` need the same change.
 
 ### 💚 Maintenance
 

README.md

@@ -115,7 +115,7 @@ There are more libraries and technologies used, of course. Docspell is
 only a orchestration of great tools and libs. One important is the
 [Stanford-NLP](https://nlp.stanford.edu/software/), that provides the
 ML features. Furthermore, file processing relies on external tools
-like tesseract, unoconv and ocrmypdf. All dependencies can be looked
+like tesseract, unoserver/unoconvert and ocrmypdf. All dependencies can be looked
 up in `project/Dependencies.scala`.
 
 

modules/convert/src/main/scala/docspell/convert/extern/Unoconv.scala

@@ -26,7 +26,7 @@ object Unoconv {
     val cmd = cfg.command.withVars(Map.empty)
 
     ExternConv.toPDF[F, A](
-      "unoconv",
+      "unoconvert",
       cmd,
       cfg.workingDir,
       useStdin = false,

modules/convert/src/test/scala/docspell/convert/ConversionTest.scala

@@ -63,8 +63,8 @@ class ConversionTest extends FunSuite with FileChecks with TestLoggingConfig {
     ),
     UnoconvConfig(
       ExternalCommand(
-        "unoconv",
-        Seq("-f", "pdf", "-o", "{{outfile}}", "{{infile}}"),
+        "unoconvert",
+        Seq("{{infile}}", "{{outfile}}", "--convert-to", "pdf"),
         Duration.seconds(20)
       ),
       target

modules/convert/src/test/scala/docspell/convert/extern/ExternConvTest.scala

@@ -55,14 +55,14 @@ class ExternConvTest extends FunSuite with FileChecks with TestLoggingConfig {
 
   test("convert office to pdf") {
     val cfg = ExternalCommand(
-      "unoconv",
-      Seq("-f", "pdf", "-o", "{{outfile}}", "{{infile}}"),
+      "unoconvert",
+      Seq("{{infile}}", "{{outfile}}", "--convert-to", "pdf"),
       Duration.seconds(20)
     )
 
     assume(commandExists(cfg.program), s"Command ${cfg.program} not found. Ignore tests.")
     File
-      .withTempDir[IO](target, "unoconv")
+      .withTempDir[IO](target, "unoconvert")
       .use(dir =>
         IO {
           val ucCfg = UnoconvConfig(cfg, target)

modules/joex/src/main/resources/reference.conf

@@ -629,22 +629,25 @@ Docpell Update Check
     }
 
     # To convert "office" files to PDF files, the external tool
-    # unoconv is used. Unoconv uses libreoffice/openoffice for
-    # converting. So it supports all formats that are possible to read
-    # with libreoffice/openoffic.
+    # unoconvert (from unoserver) is used. It talks to a running
+    # unoserver daemon, which uses LibreOffice for converting. So it
+    # supports all formats that LibreOffice can read.
     #
-    # Note: to greatly improve performance, it is recommended to start
-    # a libreoffice listener by running `unoconv -l` in a separate
-    # process.
+    # The config key remains `unoconv` for compatibility with existing
+    # overrides; the default program is now `unoconvert`.
+    #
+    # Note: start a unoserver daemon separately (Docker images do this
+    # in the joex entrypoint). Concurrent convert requests are queued
+    # by the server, which avoids LibreOffice listener deadlocks.
+    # See https://github.com/unoconv/unoserver/ and issue #3345.
     unoconv = {
       command = {
-        program = "unoconv"
+        program = "unoconvert"
         args = [
-          "-f",
-          "pdf",
-          "-o",
+          "{{infile}}",
           "{{outfile}}",
-          "{{infile}}"
+          "--convert-to",
+          "pdf"
         ]
         timeout = "2 minutes"
       }

website/site/content/docs/install/docker.md

@@ -20,7 +20,7 @@ release page. The images contain all the necessary
 
 - `docspell/restserver` this images contains the http server
 - `docspell/joex` this image contains the job executor and all
-  required software (ocrmypdf, unoconv etc) mentioned in
+  required software (ocrmypdf, unoserver/unoconvert, etc) mentioned in
   [prerequisites](@/docs/install/prereq.md).
 - `docspell/dsc` this is an image containing a
   [cli](@/docs/tools/cli.md) for docspell that can be used to watch

website/site/content/docs/install/prereq.md

@@ -46,27 +46,37 @@ component.
   images into pdf files. It is a widely used open source OCR engine.
   Tesseract 3 and 4 should work with docspell; you can adopt the
   command line in the configuration file, if necessary.
-- [Unoconv](https://github.com/unoconv/unoconv) is used to convert
-  office documents into PDF files. It uses libreoffice/openoffice.
+- [Unoserver](https://github.com/unoconv/unoserver/) provides
+  `unoconvert`, used to convert office documents into PDF files via a
+  running LibreOffice listener (`unoserver` daemon).
 - [wkhtmltopdf](https://wkhtmltopdf.org/) is used to convert HTML into
   PDF files.
 - [OCRmyPDF](https://github.com/jbarlow83/OCRmyPDF) can be optionally
   used to convert PDF to PDF files. It adds an OCR layer to scanned
   PDF files to make them searchable. It also creates PDF/A files from
   the input pdf.
 
-The performance of `unoconv` can be improved by starting `unoconv -l`
-in a separate process. This runs a libreoffice/openoffice listener and
-therefore avoids starting one each time `unoconv` is called.
+Start a `unoserver` daemon on machines that run joex (the official
+Docker joex image does this in the entrypoint). Concurrent convert
+requests are queued by the server, which avoids LibreOffice listener
+deadlocks (see [#3345](https://github.com/eikek/docspell/issues/3345)).
 
 ### Example Debian
 
-On Debian this should install all joex requirements:
+On Debian this should install most joex requirements. Install
+`unoserver` separately with the LibreOffice-compatible Python (see the
+[unoserver README](https://github.com/unoconv/unoserver/)):
 
 ``` bash
-sudo apt-get install ghostscript tesseract-ocr tesseract-ocr-deu tesseract-ocr-eng unpaper unoconv wkhtmltopdf ocrmypdf
+sudo apt-get install ghostscript tesseract-ocr tesseract-ocr-deu tesseract-ocr-eng unpaper libreoffice wkhtmltopdf ocrmypdf
+sudo -H pip install unoserver
 ```
 
+Then start the daemon before joex, for example:
+
+``` bash
+unoserver &
+```
 # Apache SOLR
 
 SOLR is a very powerful fulltext search engine and can be used to

website/site/content/docs/joex/file-processing.md

@@ -154,12 +154,14 @@ as described above.
 
 ### Office
 
-To convert office files, [Libreoffice](https://www.libreoffice.org/)
-is required and used via the command line tool
-[unoconv](https://github.com/unoconv/unoconv).
-
-To improve performance, it is recommended to start a libreoffice
-listener by running `unoconv -l` in a separate process.
+To convert office files, [LibreOffice](https://www.libreoffice.org/)
+is required and used via
+[unoserver](https://github.com/unoconv/unoserver/) / `unoconvert`.
+
+Start a `unoserver` daemon separately (Docker joex images do this in
+the entrypoint). The server queues convert requests, which avoids
+LibreOffice deadlocks when joex runs with `scheduler.pool-size > 1`
+(see [#3345](https://github.com/eikek/docspell/issues/3345)).
 
 
 ### PDF