Deploying on OpenBSD

Recently with some core changes happening to TLGS (the search engine I run on Gemini), deployment also moved from Ubuntu to OpenBSD -- for many reasons. Northwire Mail still runs on standard Linux because that's where I have the infra for serious data backup. But hardening applictions on Linux is a nightmare. It took me days, even with the help of LLMs, to figure out what exact syscalls it needs and what can be moved before sandbox enforcement (ex: OpenSSL lazy initializes so I need to warm it up and pray the behaviour doesn't change). Plus it's just a lot tricks and code to merely mimic what OpenBSD does by default. Moving to OpenBSD seems the righr decision!

Well, is not all sunshine and rainbows. There's some issues that I have to overcome. Ultimately I think the benefit far outweighs the cons.

Mitigations and default security

You can read my previous article on what is possible on Linux if you try. Linux can do surprisingly well if you take care enabling and even coding your own mitigations. While it works, coding your own security mechanism is not advisable and it is really the OS's job to make the environment secure.

Most of the mitigations described are implemented and enabled in OpenBSD by default:

• Better compiler flags (with software return integrity!)
• Hardened allocator
• Much easier to use self-sandboxing
• New libc/libcrypto/kernel each boot
• Execute only .text
• W^X by default
• Immutable object mappings

Beyond that, OpenBSD does what Linux can't (yet)

• Pinned stack address range
• Pinned syscall origins
• No /proc

The easy to use sandbox is a plus in my opinion even if your security team or LLM thinks it's a minus. Entering a good sandbox using pledge/unveil() is ~20 lines of code while using landlock and seccomp easily blows over 250. While you can implement a tighter sandbox on Linux, you won't in practice due to a combination of time pressure, there's just too many ways to mess up and figure out the exact number of surprising ways how glibc, NSS, OpenSSL and other low level components uses kernel feature.

At the end, you'll likely just set a loose set of syscall/paths to remove the most dangerous syscalls and call it a day -- to avoid preventable downtime. In contrast, with pledge/unveil, the entire system is built around their existence. The dns pledge actually is enough to get you DNS resolution. Instead of guessing which signal, path, IO syscalls and shared memory the resolver needs.

Issues and solutions

This is where I get to the not sunshine and rainbows part. There's several things I had to sort out on OpenBSD. Some of which I wish is not even an issue and some I actually appreciate the little tweaks OpenBSD developers put into the system.

PostgreSQL

By default OpenBSD sets kern.seminfo.* way too low for a PostgreSQL server to run properly. I had to follow openbsdhandbook's recommendation. Maybe it's not surprising to older sysadmins, but it was the first time for me that running a well known production software needs system setting changes.

kern.seminfo.semmni=256
kern.seminfo.semmns=2048

I knew OpenBSD has its own partition table and it's not as flexible as LVM. But I did not realize until too late, how small /var is by default. For a small VPS there's no default free space. Thus I had to go back to installation, redo manual partitioning and create a dedicated partition for /var/postgresql.

RC and cron

I starting doing serious systems work right when Ubuntu (was using Ubuntu at the time) switched to systemd and almost never used older init systems besides a short period of managing a cluster of Gentoo machines at my university.

Surprisingly RC was rather easy to use. You do need to create the service user yourself with doas useradd _appuser. The RC script itself is a few variables then RC handles the rest of the problem itself.

#!/bin/ksh

daemon="/usr/local/bin/myapp"
daemon_user="_appuser"
daemon_execdir="/var/myapp"

rc_bg=YES # Since the server doesn't daemon itself

# Standard RC settings
. /etc/rc.d/rc.subr
rc_cmd "$1"

Without the systemd daemon-reload headache and footgun! With your rc script under /etc/rc.d, then the standard management command works.

rcctl enable myapp
rcctl start myapp

Though, unlike systemd, there's no user units thus one can't deploy application without root access. However that does not stop one from running the service as a non-root user easily with the daemon_user variable.

Timers are a different problem. Systemd allows one to trigger an unit periodically. I mostly use timers to trigger periodic maintenance or update (which are most likely one shot units). RC does not have a timer feature. However, there are user crons. Edit with

# as the intended running user
VISUAL=vim crontab -e

Then allow the user to use crontab

doas ksh -c 'echo _myapp >> /var/cron/cron.allow'

Networking and pf

OpenBSD is not kidding when they mean security. Even web servers (I'm using Caddy) are not allowed to bind to port 80/443 under default settings. No.. they bind to a high port on loopback addresses. Then you set up pf to redirect. For me, my Caddy config looks like

{
    default_bind 127.0.0.1 [::1]
    http_port 24442
    https_port 24443

    ... # Other settings
}

Then in pf.conf, redirect port 80 and 443 to the high port

# Redirect standard ports to custom ports used by Caddy
pass in on egress inet proto tcp to (egress) port 80 \
    rdr-to 127.0.0.1 port 24442
pass in on egress inet proto tcp to (egress) port 443 \
    rdr-to 127.0.0.1 port 24443
pass in on egress inet6 proto tcp to (egress) port 80 \
    rdr-to ::1 port 24442
pass in on egress inet6 proto tcp to (egress) port 443 \
    rdr-to ::1 port 24443

This is quite nice actually, despite being annoying. Yeah! I shouldn't bind to privileged ports if I am not root. And I shouldn't be root at any time. Nor should there be magic settings/tools that put holes into privileged ports such that anyone on the system can listen on them (looking at you Ubuntu).

Conclusion

Otherwise OpenBSD feels just like a standard UNIX(-like) system. It runs standard POSIX APIs, standard shell and it's not like most high level language cares. Give it a shot. You might like it. And the built-in security might save you some trouble in the future.