diff --git a/Changelog.md b/Changelog.md index 27c86a66fb..85f929496d 100644 --- a/Changelog.md +++ b/Changelog.md @@ -20,6 +20,13 @@ - Improve performance of custom field statistics on large databases by scoping `custom_field_value` lookups to matching items only. +- Migrate office conversion from deprecated `unoconv` to + [unoserver](https://github.com/unoconv/unoserver/) / `unoconvert` + (#3345, #3293). The HOCON key `docspell.joex.convert.unoconv` is + kept for compatibility; the default program/args now call + `unoconvert`. **Breaking:** joex must run a `unoserver` daemon + (official Docker images start it in the entrypoint). Custom images + that relied on `unoconv -l` need the same change. ### 💚 Maintenance diff --git a/README.md b/README.md index 268b01d8bb..3354f1cc23 100644 --- a/README.md +++ b/README.md @@ -115,7 +115,7 @@ There are more libraries and technologies used, of course. Docspell is only a orchestration of great tools and libs. One important is the [Stanford-NLP](https://nlp.stanford.edu/software/), that provides the ML features. Furthermore, file processing relies on external tools -like tesseract, unoconv and ocrmypdf. All dependencies can be looked +like tesseract, unoserver/unoconvert and ocrmypdf. All dependencies can be looked up in `project/Dependencies.scala`. diff --git a/modules/convert/src/main/scala/docspell/convert/extern/Unoconv.scala b/modules/convert/src/main/scala/docspell/convert/extern/Unoconv.scala index 102987d25b..73e224af43 100644 --- a/modules/convert/src/main/scala/docspell/convert/extern/Unoconv.scala +++ b/modules/convert/src/main/scala/docspell/convert/extern/Unoconv.scala @@ -26,7 +26,7 @@ object Unoconv { val cmd = cfg.command.withVars(Map.empty) ExternConv.toPDF[F, A]( - "unoconv", + "unoconvert", cmd, cfg.workingDir, useStdin = false, diff --git a/modules/convert/src/test/scala/docspell/convert/ConversionTest.scala b/modules/convert/src/test/scala/docspell/convert/ConversionTest.scala index cfbd08374e..b2d81ffdc0 100644 --- a/modules/convert/src/test/scala/docspell/convert/ConversionTest.scala +++ b/modules/convert/src/test/scala/docspell/convert/ConversionTest.scala @@ -63,8 +63,8 @@ class ConversionTest extends FunSuite with FileChecks with TestLoggingConfig { ), UnoconvConfig( ExternalCommand( - "unoconv", - Seq("-f", "pdf", "-o", "{{outfile}}", "{{infile}}"), + "unoconvert", + Seq("{{infile}}", "{{outfile}}", "--convert-to", "pdf"), Duration.seconds(20) ), target diff --git a/modules/convert/src/test/scala/docspell/convert/extern/ExternConvTest.scala b/modules/convert/src/test/scala/docspell/convert/extern/ExternConvTest.scala index 67d1e5ae76..4fa6a415b2 100644 --- a/modules/convert/src/test/scala/docspell/convert/extern/ExternConvTest.scala +++ b/modules/convert/src/test/scala/docspell/convert/extern/ExternConvTest.scala @@ -55,14 +55,14 @@ class ExternConvTest extends FunSuite with FileChecks with TestLoggingConfig { test("convert office to pdf") { val cfg = ExternalCommand( - "unoconv", - Seq("-f", "pdf", "-o", "{{outfile}}", "{{infile}}"), + "unoconvert", + Seq("{{infile}}", "{{outfile}}", "--convert-to", "pdf"), Duration.seconds(20) ) assume(commandExists(cfg.program), s"Command ${cfg.program} not found. Ignore tests.") File - .withTempDir[IO](target, "unoconv") + .withTempDir[IO](target, "unoconvert") .use(dir => IO { val ucCfg = UnoconvConfig(cfg, target) diff --git a/modules/joex/src/main/resources/reference.conf b/modules/joex/src/main/resources/reference.conf index 08121aab9f..1f3941255f 100644 --- a/modules/joex/src/main/resources/reference.conf +++ b/modules/joex/src/main/resources/reference.conf @@ -629,22 +629,25 @@ Docpell Update Check } # To convert "office" files to PDF files, the external tool - # unoconv is used. Unoconv uses libreoffice/openoffice for - # converting. So it supports all formats that are possible to read - # with libreoffice/openoffic. + # unoconvert (from unoserver) is used. It talks to a running + # unoserver daemon, which uses LibreOffice for converting. So it + # supports all formats that LibreOffice can read. # - # Note: to greatly improve performance, it is recommended to start - # a libreoffice listener by running `unoconv -l` in a separate - # process. + # The config key remains `unoconv` for compatibility with existing + # overrides; the default program is now `unoconvert`. + # + # Note: start a unoserver daemon separately (Docker images do this + # in the joex entrypoint). Concurrent convert requests are queued + # by the server, which avoids LibreOffice listener deadlocks. + # See https://github.com/unoconv/unoserver/ and issue #3345. unoconv = { command = { - program = "unoconv" + program = "unoconvert" args = [ - "-f", - "pdf", - "-o", + "{{infile}}", "{{outfile}}", - "{{infile}}" + "--convert-to", + "pdf" ] timeout = "2 minutes" } diff --git a/website/site/content/docs/install/docker.md b/website/site/content/docs/install/docker.md index 6033244c09..dc1cea222c 100644 --- a/website/site/content/docs/install/docker.md +++ b/website/site/content/docs/install/docker.md @@ -20,7 +20,7 @@ release page. The images contain all the necessary - `docspell/restserver` this images contains the http server - `docspell/joex` this image contains the job executor and all - required software (ocrmypdf, unoconv etc) mentioned in + required software (ocrmypdf, unoserver/unoconvert, etc) mentioned in [prerequisites](@/docs/install/prereq.md). - `docspell/dsc` this is an image containing a [cli](@/docs/tools/cli.md) for docspell that can be used to watch diff --git a/website/site/content/docs/install/prereq.md b/website/site/content/docs/install/prereq.md index 6111b18083..9ae7364742 100644 --- a/website/site/content/docs/install/prereq.md +++ b/website/site/content/docs/install/prereq.md @@ -46,8 +46,9 @@ component. images into pdf files. It is a widely used open source OCR engine. Tesseract 3 and 4 should work with docspell; you can adopt the command line in the configuration file, if necessary. -- [Unoconv](https://github.com/unoconv/unoconv) is used to convert - office documents into PDF files. It uses libreoffice/openoffice. +- [Unoserver](https://github.com/unoconv/unoserver/) provides + `unoconvert`, used to convert office documents into PDF files via a + running LibreOffice listener (`unoserver` daemon). - [wkhtmltopdf](https://wkhtmltopdf.org/) is used to convert HTML into PDF files. - [OCRmyPDF](https://github.com/jbarlow83/OCRmyPDF) can be optionally @@ -55,18 +56,27 @@ component. PDF files to make them searchable. It also creates PDF/A files from the input pdf. -The performance of `unoconv` can be improved by starting `unoconv -l` -in a separate process. This runs a libreoffice/openoffice listener and -therefore avoids starting one each time `unoconv` is called. +Start a `unoserver` daemon on machines that run joex (the official +Docker joex image does this in the entrypoint). Concurrent convert +requests are queued by the server, which avoids LibreOffice listener +deadlocks (see [#3345](https://github.com/eikek/docspell/issues/3345)). ### Example Debian -On Debian this should install all joex requirements: +On Debian this should install most joex requirements. Install +`unoserver` separately with the LibreOffice-compatible Python (see the +[unoserver README](https://github.com/unoconv/unoserver/)): ``` bash -sudo apt-get install ghostscript tesseract-ocr tesseract-ocr-deu tesseract-ocr-eng unpaper unoconv wkhtmltopdf ocrmypdf +sudo apt-get install ghostscript tesseract-ocr tesseract-ocr-deu tesseract-ocr-eng unpaper libreoffice wkhtmltopdf ocrmypdf +sudo -H pip install unoserver ``` +Then start the daemon before joex, for example: + +``` bash +unoserver & +``` # Apache SOLR SOLR is a very powerful fulltext search engine and can be used to diff --git a/website/site/content/docs/joex/file-processing.md b/website/site/content/docs/joex/file-processing.md index 5a4da05349..3c43284e9c 100644 --- a/website/site/content/docs/joex/file-processing.md +++ b/website/site/content/docs/joex/file-processing.md @@ -154,12 +154,14 @@ as described above. ### Office -To convert office files, [Libreoffice](https://www.libreoffice.org/) -is required and used via the command line tool -[unoconv](https://github.com/unoconv/unoconv). - -To improve performance, it is recommended to start a libreoffice -listener by running `unoconv -l` in a separate process. +To convert office files, [LibreOffice](https://www.libreoffice.org/) +is required and used via +[unoserver](https://github.com/unoconv/unoserver/) / `unoconvert`. + +Start a `unoserver` daemon separately (Docker joex images do this in +the entrypoint). The server queues convert requests, which avoids +LibreOffice deadlocks when joex runs with `scheduler.pool-size > 1` +(see [#3345](https://github.com/eikek/docspell/issues/3345)). ### PDF