Flotilla

An easily deployed server based on Docker compose, complete with privacy tools, a media center, cloud storage, and much more.

Currently it is designed with Fedora Server in mind with SELinux set to enforcing. The Flotilla is protected by Suricata, an intrusion prevention system.

Installation

There are a handful of different configurations available. alpha is the overarching one which contains nearly all services. devops is oriented for production of solely code. Others may be created at a later time.

Simply run ./flotilla install [alpha | devops] to install Flotilla. From now on you may access Flotilla's commands from all contexts (flotilla status, no more ./)

If you have NAS storage, you can symlink /opt/flotilla/data to the mount point.

Credentials and Secrets

Credentials are stored in /opt/flotilla/secrets.env, a template exists in the root of this repository. Fill it out so that docker containers may read them.

Unfortunately many containers have yet to accept the standard Secrets feature recently added in Docker.

Quirks

Docker will complain with this error on some versions of Fedora:

ERROR: for CONTAINER_NAME  Cannot start service CONTAINER_NAME: open /dev/dma_heap: permission denied

As of writing Fedora 33 has an SELinux issue where /dev/dma_heap/ gets labeled incorrectly. Follow these commands to fix it:

sudo setenforce 0
# Check it with:
# ls -aZ /dev/dma*
sudo chcon -t device_t /dev/dma_heap
sudo setenforce 1

Services

Some special manual actions are currently are necessary to get a handful of services running.

Actual Budget

The alpha fleet includes Actual Budget at https://actual.maxocull.com. SWAG terminates HTTPS and proxies to the actual container on port 5006. The proxy provides the cross-origin isolation headers required by Actual's browser database and inherits SWAG's WebSocket forwarding. The host port is bound only to 127.0.0.1, not exposed directly to the network.

Actual stores its server configuration, budget files, and bank-sync tokens in /opt/flotilla/data/actual, mounted at /data with a private SELinux label. Protect this directory and its backups as sensitive financial data. Actual uses its own server password; no new plaintext password belongs in Compose or Git.

For an existing installation, copy the updated alpha/docker-compose.yml to /opt/flotilla/docker-compose.yml and copy config/letsencrypt/nginx/proxy-confs/actual.subdomain.conf into the matching /opt/flotilla/config/letsencrypt/nginx/proxy-confs/ directory. Start Actual:

docker compose -f /opt/flotilla/docker-compose.yml up -d actual
# From your workstation, replace fleet-host with your SSH host:
ssh -L 5006:127.0.0.1:5006 fleet-host

Open http://localhost:5006 through that tunnel and set the Actual server password before publishing its hostname. Then add DNS for actual.maxocull.com pointing to this fleet host. The SWAG certificate subdomain list includes actual; recreate SWAG to request the updated certificate and load the proxy:

docker compose -f /opt/flotilla/docker-compose.yml up -d letsencrypt
docker compose -f /opt/flotilla/docker-compose.yml exec letsencrypt nginx -t

Connect the finance CLI with ACTUAL_SERVER_URL=https://actual.maxocull.com, the server password in ACTUAL_PASSWORD, and the budget's advanced-settings sync ID in ACTUAL_SYNC_ID. If budget encryption is enabled, configure ACTUAL_ENCRYPTION_PASSWORD separately. Keep these values in the finance project's private .env, outside Git. Actual remains responsible for bank sync; SimpleFIN setup and reauthentication happen in Actual's UI.

The image is pinned to actualbudget/actual-server:26.9.0 and excluded from Watchtower updates. Back up the entire data directory with the service stopped before changing its version; update the finance project's @actual-app/api to a compatible version as part of the same upgrade. Restore the directory to recover server settings and budgets. The existing flotilla borg backup command backs up Nextcloud only and does not cover Actual automatically. Generated finance workbooks are not a substitute for an Actual server backup.

Suricata

The entire swarm is defended by Suricata, an Intrusion Prevention System (IPS) which runs on the host and intercepts malicious packets before they make contact with containers (or any other service on the host). Suricata automatically updates rules using a cron job which is performed daily.

Suricata requires some IPTables rules to intercept these packets, so be warned that while these rules are enabled, Suricata must also be online; otherwise, no data will pass, and any connections (e.g., SSH) will be terminated.

The IPS may be handled via the flotilla ips subcommands.

If you have issues, check that /etc/sysconfig/suricata is using the -q 0 option so that it can execute in inline (IPS) mode and intercept packets! It must also be running in repeat mode (with masks), or else it will skip all other filters (effectively making your firewall useless).

FirewallD

FirewallD is used to intelligently manage IPTables. It will not manage Docker services due to it's independent firewall, and it does not manage Suricata.

Firewall commands are accessible via flotilla fw subcommands.

Nginx & Let's Encrypt

All that should be required other than tweaking the docker-compose file is adding /opt/flotilla/config/letsencrypt/nginx/.htpasswd:

# The -C <level> is for bcrypt, you can make it stronger by increasing it up to 17.
htpasswd -B -C 10 -c .htpasswd <user> # Then fill in the password prompt.

# For additional users:
htpasswd -B .htpasswd <user> # Then fill in the password prompt.

qBittorrent

qBittorrent's web panel is restricted to local-only access. It's default username and password are admin and adminadmin respectively. It's recommended you log in and change them (Tools > Options > Web UI) if you don't bypass the login with whitelisted IP's.

In order for this to work using Sonarr et. al., under Tools > Options > Downloads set Default Save Path to /data. Set Monitored Folder to /torrents.

It's recommened you enable Anonymous mode under Tools > Options > BitTorrent, and optionally bump up your queueing and enable not counting slow torrent.

Leave the fixed port (6881) alone, it will break otherwise.

Sonarr, Radarr, Lidarr

Enable Advanced Settings, then under Settings > Indexers add a Torznab entry with at least these settings:

Name: Jackett All
Enable RSS Sync: Yes
Enable Search: Yes
URL: http://jackett:9117
API Path: /torznab/all/api
API Key: <Get this from the Jackett web terminal>

This should track all available indexes in Jackett.

Then under Settings > Download Client, add an entry with at least these settings:

Name: Flotilla qBittorrent
Enable: Yes
Host: cleanroom
Port: 8800
Username: admin         # Unless changed.
Password: adminadmin    # Unless changed.
Use SSL: No

Connect your download client folder under Settings > Download Client > Remote Path Mappings by filling in these settings:

Host: cleanroom
Remote Path: /data/
Local Path: /data/

When you add a new {Series,Movie,Song}, you'll want to set the path it gets placed in to {/tv/,/movies/,/music/} respectively.

If this is exposed to the internet, you'll likely want to add authentication as well, which is configurable under Settings > General > Security. You'll likely also want to turn some quality settings down since they start on unlimited.

Jellyfin

Add Libraries for the following internal, shared volumes:

• /data/tvshows as TV Shows from Sonarr
• /data/movies as Movies from Radarr
• /data/music as Music from Lidarr
• /data/books as Books from Calibre?
• /data/photos as Photos from various uploads

If you see an endlessly spinning circle on the webpage, clear your browser cache for the page.

Optionally enable hardware accelerated transcoding under Dashboard > Playback > Transcoding, VAAPI-enabled devices should already be mounted into the Jellyfin container.

Postgres & PGAdmin

Login with the email and password you specified in the config files.

Click Add New Server and fill out:

[General]
Name: Flotilla Postgres

[Connection]
Host name/address: postgres
Password: <POSTGRES_PASSWORD from secrets.env>
Save Password: true

The user ID must be 1000 in order to share the key files from letsencrypt.

For security, each user is restricted to a table in pg_hba.conf, you will have to add new users under there.

Nextcloud

Create a database for Nextcloud inside of Postgres:

CREATE USER nextcloud WITH PASSWORD 'password';
CREATE DATABASE nextcloud TEMPLATE template0 ENCODING 'UNICODE';
ALTER DATABASE nextcloud OWNER TO nextcloud;
GRANT ALL PRIVILEGES ON DATABASE nextcloud TO nextcloud;

password should match the one provided for POSTGRES_PASS in secrets.env.

Any extra domains must be added to config/nextcloud/config.php in addition to a few extra settings like so:

  'trusted_proxies' =>
  array (
    0 => 'letsencrypt',
  ),
  'trusted_domains' =>
  array (
    0 => 'cloud.maxocull.com',
    1 => 'cloud.maxocull.net',
  ),
  'overwriteprotocol' => "https",
  'overwritehost' => "cloud.maxocull.com",
  'overwrite.cli.url' => 'https://cloud.maxocull.com/',

Gitlab

Create a database for Gitlab inside of Postgres:

CREATE ROLE gitlab with LOGIN CREATEDB PASSWORD 'password';
CREATE DATABASE gitlabhq_production;
GRANT ALL PRIVILEGES ON DATABASE gitlabhq_production to gitlab;

password should match the one provided for DB_PASS in secrets.env. This use should have permission to create databases, but he does not have to be superuser. Afterward, create the pg_trgm extension on the database with the superuser (who should be postgres).

Since the host will be using port 22, this container will share it by creating a git user on the host and SSH tunnelling into the Docker network's Gitlab instance. This requires symlinking manually copying SSH keys from the container volume bind mount into the local git user's .ssh folder. See here for more details. It must be manually copied because of SELinux permissions; container security contexts may not mix with sshd contexts of a different user.

The Gitlab Runner must be connected to the Gitlab instance. Paste the key from the web UI into the REGISTRATION_TOKEN environment variable.

Gitlab must also connect to a Docker registry to host Docker images. This shares the letsencrypt certificates to make API calls.

TODO: Automate this procedure. TODO: Automate update of SSH keys, possibly with inotify/crond/incrontab.

Heimdall

Any services you add which contact an API will require the hostname of the Docker container and their port listed. As an example, for qBittorrent you would enter:

URL: cleanroom:8800
Username: admin         # Unless changed.
Password: adminadmin    # Unless changed.

LazyLibrarian

Some configuration options should be set.

• Config > Interface > Access Control: Set a WebServer login
• Config > Interface > Startup: Enable API
• Config > Interface > Appearance: Set to flatly
• Config > Interface > OPDS Server: Enable, require credentials, leave credentials empty, enable metadata
• Config > Downloaders > Torrents: Enable qBittorrent, cleanroom:8800, admin and adminadmin (unless changed), /downloads
• Config > Providers > Torznab Providers: Name it Jackett All, enable, http://jackett:9117/torznab/all, enter the API key from the Jackett webpage
• Config > Providers > Direct Download Providers: You may have to change the mirror (libgen.pw, gen.lib.rus.ec, ...), set one to use search.php and the other foreignfiction/index.php, enable Z-Library as well with b-ok.cc
• Config > Processing > Calibre: Set the calibredb import program to /usr/bin/calibredb

Calibre

Setup LazyLibrarian first to generate a databse. Otherwise, you must manually generate a metadata.db file and upload it to /books using Calibre desktop. It is unfortunate that the main project does not support automatically generating this file.

At the setup menu, enter these values:

Location of Calibre databse: /books
Enable uploading: true
Use Calibre's ebook converter: true
Path to convertertool: /usr/bin/ebook-convert
Location of Unrar binary: /usr/bin/unrar

The default login is admin and admin123. Change these.

Mayan EDMS

Mayan needs a database, create one in our existing Postgres DB. First make a user mayan with the password you specified ($MAYAN_DATABASE_PASSWORD), then make a database named mayan owned by the mayan user.

If after first run it doesn't provide a web dialog with a password, the initialization failed. Try again by running: docker-compose exec mayan /opt/mayan-edms/bin/mayan-edms.py initialsetup.

Configure the email account:

Label: maxocull.com Gmail
Default: true
Enabled: true
Host: smtp.gmail.com
Port: 587
Use TLS: true
Use SSL: false
Username: maxocull.com
Password: <Gmail password, does not need an app password, should match $SMTP_PASSWORD>
From: max.ocull@gmail.com

Kimai

You'll need to touch this file:

touch /opt/flotilla/config/kimai/local.yaml:

BigBlueButton and Greenlight

Create a database for Greenlight inside of Postgres:

CREATE USER greenlight WITH PASSWORD 'password';
CREATE DATABASE greenlight_production;
ALTER DATABASE greenlight_production OWNER TO greenlight;
GRANT ALL PRIVILEGES ON DATABASE greenlight_production TO greenlight;

password should match the one provided for DB_PASSWORD in secrets.env.

You'll need to generate secret keys for BBB and Greenlight and place them in secrets.env.

Wireguard

Fedora does not load the wireguard module by default. To load it on boot, create a file at /etc/modules-load.d/wireguard.conf with contents:

wireguard

Then run sudo modprobe wireguard to load it without rebooting.

TODO: Document config process.

Borg Off-site Backups

Flotilla can back up host workloads to an off-site Debian host named bastion over SSH. bastion stores the Borg repository at /mnt/backup/repos/entourage; the source host does not stage tarballs, SQL dumps, or local archives. Today the first workload is Entourage's Docker Compose Nextcloud installation, but the repository setup is intentionally separate from that workload so more backup jobs can be added later.

Required non-committed secrets:

• /opt/flotilla/secrets/borg_passphrase
• /opt/flotilla/secrets/entourage_borg_ssh on Entourage
• /opt/flotilla/secrets/entourage_borg_ssh.pub on bastion

Provision bastion after installing Debian:

./flotilla borg install-bastion

The bastion installer installs Borg and OpenSSH; creates the borg user; initializes the encrypted repository with repokey-blake2; installs a restricted append-only authorized_keys entry for Entourage; and enables local monthly borg check, borg prune, and borg compact timers.

Provision Entourage:

./flotilla borg install-entourage

Then edit /opt/flotilla/config/borg/entourage.env if BORG_BACKUP_HOST=bastion is not the right SSH hostname or IP address. If secrets were not present during install, enable the timer after adding them with systemctl enable --now flotilla-borg-nextcloud-backup.timer. The weekly backup timer stops the configured Nextcloud containers, defaults to nextcloud and nextcloud-cron, leaves PostgreSQL running, streams pg_dump directly into Borg as postgres/nextcloud.dump, backs up the Nextcloud bind mounts and compose configuration directly, then restarts the stopped Nextcloud containers. Archive creation uses zstd,1 compression and prints Borg progress in the systemd journal.

The backup runs from systemd as root, so it does not use a normal user's ~/.ssh/config. Put SSH routing details for bastion in /opt/flotilla/config/borg/ssh_config; for example, set HostName to a local DNS name, a private FQDN, or an IP address reachable from Entourage. The Entourage installer runs ssh-keyscan against that HostName and stores the result for the bastion alias in /opt/flotilla/config/borg/known_hosts.

Installed systemd units:

• flotilla-borg-nextcloud-backup.timer: runs the Entourage Nextcloud backup weekly on Thursday at 00:30 America/Indiana/Indianapolis time.
• flotilla-borg-nextcloud-backup.service: one-shot service invoked by the backup timer; it runs on Entourage and creates a Borg archive on the repository server using zstd,1 compression.
• flotilla-borg-repo-check.timer: runs repository integrity checks monthly on bastion.
• flotilla-borg-repo-check.service: one-shot service invoked by the check timer; it runs borg check against the local repository.
• flotilla-borg-repo-prune-compact.timer: runs retention cleanup monthly on bastion.
• flotilla-borg-repo-prune-compact.service: one-shot service invoked by the prune/compact timer; it runs borg prune with the configured retention policy and then borg compact.

Run the same job for the initial local seed and future off-site incrementals:

flotilla borg backup

Useful restore commands from Entourage:

source /opt/flotilla/config/borg/entourage.env
export BORG_PASSPHRASE="$(<"${BORG_PASSPHRASE_FILE}")"
export BORG_RSH
borg list "${BORG_BACKUP_USER}@${BORG_BACKUP_HOST}:${BORG_REPO_PATH}"
borg mount "${BORG_BACKUP_USER}@${BORG_BACKUP_HOST}:${BORG_REPO_PATH}::ARCHIVE" /mnt/borg

Restore single files by copying them out of the mounted archive. For a full Nextcloud restore, stop Nextcloud, restore /opt/flotilla/config/nextcloud, /opt/flotilla/data/nextcloud, and /opt/flotilla/docker-compose.yml, then restore the database dump with pg_restore into the Nextcloud PostgreSQL database before starting Nextcloud again.

TODO List

• NordVPN router
• Switching to Deluge
• Set Jackett to use VPN rather than proxy
• Shadowsocks proxy server through the VPN (does this gain me anything?)
• Matomo/Piwik for analytics
• A TOR node, then hosting an onion site
• An IRC server