The Source Will Be With You, Always

Thoughts and News

Problems with Systemd and Why I like BSD Init


In my first post on this blog, I said that systemd was one of the things that pushed me to look into the BSDs and eventually move everything to FreeBSD. People have asked me what my problem with systemd is, but I haven't written about it. So finally, here are my thoughts.

I've been reluctant to discuss systemd there because I was sick of everyone fighting over it. The Gentoo mailing list (and everyone else on the internet) became a giant flame war where everyone either thinks that systemd is evil and attacks Lennart Poettering personally, or thinks that systemd is going to save us all from ourselves and everyone must use it. I was just sick of everyone being so unreasonable.

For my part, I'm not a fan of systemd but I also don't think it's the end of the world. I watched a great interview with Lennart on the Linux Action Show about why he implemented it, and he had some good reasons. To write a daemon for Linux, you need to maintain a different init script for each distro because they all put things in different places. And sysVinit isn't the best with dependencies. Developing things for the Linux desktop is not as easy as it could be, due in large part to the fragmentation.

I avoided it when I was running Linux at home, but these days I have to use Linux for work and can't avoid it any longer. After using systemd for a while, there are definitely some things I like about it. It makes easy things easier. Like taking some daemon I wrote and making it a service. The service files are nice, and telling it keep restarting my service if it crashes is super handy. I like that the service file will work on any Linux distro with systemd. I can just about write a service file from memory, whereas with an rc script I'll always need to start from an example.

On the other hand, it makes the hard things even harder. If I want to do something complex that systemd doesn't have an option for, I'm out of luck. With BSD init, I can just edit the /etc/rc script and replace it with arbitrary shell code. By default it looks under /etc/rc.d/ and runs the enabled startup scripts there, but I can change that in less than five minutes.

Here's FreeBSD's /etc/rc. /sbin/init calls this script, and /etc/rc calls all the enabled scripts under /etc/rc.d/.


Notice how tiny it is. I can change anything in here I want. I can remove the part where it starts services and manually start just one or two services. I can change where the console output goes, like redirecting it to a file. I can make this script call whatever I like, however I like. Unlimited flexibility.

I'm not an expert in init systems, but I used OpenRC for a long time and I'm pretty familiar with BSD init now. From what I understand, BSD init is superior to the old sysVinit that Linux used to use. And Gentoo's OpenRC is similar to BSD init. I don't have much experience with classic sysVinit. So that's where I'm coming from.

Systemd is very opaque. What I mean by that is that I can't easily debug it. I can't look inside and see what it's doing, because it's a complicated binary. I can't easily make changes to the way it behaves. BSD init is just a script, so I can look at it, add print statements, and change it in any way I like. It's easy to look at it and understand exactly what it's going to do and why.

The way systemd does parallel startup bothers me. If service A depends on service B, they were traditionally started in sequence; A only starts after B is done starting. Systemd will start them both at the same time, but buffer any messages A tries to send to B until B is ready. So A just thinks that B is being slow to respond, and system startup is faster. You can disable this for certain things by using the 'After' directive, but that doesn't apply to the whole system and anyhow, disabling a feature isn't an answer to whether it should exist in the first place.

My problem with this is that the order in which services are started should, in my opinion, be exactly the same each time and predictable to the sysadmin. With systemd, the order is not deterministic, so you don't know what's going to happen next time you boot. I work with servers and embedded devices; I don't care much about boot time. A server spends several minutes in the BIOS during POST anyway, before the bootloader is even run; making the OS boot faster doesn't change very much. Embedded devices already start quickly because you trim them down to the bare minimum. What I care about is that every time I boot, the same exact things happen in the same exact order -- the order that I want.

It seems no one can agree on whether systemd is modular or not. I think the problem is with different definitions of 'modularity'. Systemd doesn't put everything in PID 1 like some people suggest; it uses modules that communicate with each other. So it is modular in that sense. But these modules are very tightly integrated. You can't easy remove some of them, or replace them with other things. So in that sense it is very monolithic. This is not at all like having a simple interface and passing data via stdin and stdout, which is the modularity that makes UNIX pipes possible. This is the sense that matters to me.

I dislike the way systemd is absorbing everything. It's not just an init system, it's become an everything-under-the-hood includes-the-kitchen-sink management system. That doesn't feel modular to me. Why should systemd implement NTP when ntpd already exists? I think systemd-timesyncd and all the others like it are just reinventing the wheel.

In the case of ntpd, you might say that it's a monstrosity that needs to be replaced with something much smaller, and you'd be right. There are others re-implementing it. OpenBSD created OpenNTPD, which is very lightweight, client-only, and can even hit HTTPS sites to verify that the times it gets from NTP servers is reasonable. Cleaning up or re-implementing ntpd is a good thing, but OpenBSD didn't feel the need to make it part of init.

EDIT 2015-10-31: Thanks to @myfreeweb for correcting me that OpenNTPD is not client-only. It is, however, client-oriented, as it sacrifices some accuracy for simplicity. If you're running a time server, you'll probably be using something else.

Linux is very fragmented and systemd is bringing things back together a bit. That may be a good thing. But the way we're headed, there will come a point when we won't be able to say 'GNU/Linux' anymore. It'll be 'GNU/systemd-linuxd'.

But part of what has made Linux so special is the ability to swap out literally any component on the system with something else -- even libc! All a Linux program needs is a kernel and something that looks like a libstdc.so. It's common to build embedded Linux devices with ulibc, or something else tiny. BSD can't just replace its libc, it's too integrated.

There was a really good interview with some GNOME devs on BSD Now not long ago. They said they were going to pay more attention to BSD compatibility, but also talked about why they needed to depend on systemd: There is no API or standard way to do things like set the timezone, change wifi networks, or perform many common desktop tasks on Linux. GNOME's code is full of #ifdefs so that they can use different code for each distro or OS, and it's not maintainable. Systemd provides these APIs in a clean and cross-platform manner, which is a big win for desktop environments.

OpenBSD is working on systemBSD, which is a systemd shim. It provides the APIs that GNOME needs, but doesn't implement the rest of systemd. I think having APIs like this is a really good thing, and I'd like to see us as an industry move toward standardizing on it (or something like it).

Systemd still feels unstable. I'm not sure whether it's because it actually has more problems than other software or just because people are quick to publicize its flaws, but it's had some problems that would be really funny if they weren't so serious. I've included links to the actual bug reports:

For a critical piece of infrastructure, having regular bugs like this is a big problem. It renders machines unbootable and ruins people's days. Compare this to a really well-managed project like OpenZFS. To my knowledge, no one has ever lost data as a result of a bug in ZFS. Infrastructure projects like this have to be held to a higher standard than most software because the consequences for a bug can be so severe.

I think the systemd devs suffer from NIH syndrome. I can't find anything to suggest that they considered OpenRC; they seem to have a personal problem with Gentoo. Many of the problems they described with init systems are already solved by both OpenRC and BSD init. Now they're going around re-implementing every possible Linux daemon as a systemd module, just because they can.

I was talking to a reader recently who said, "Every time I look at systemd development, I get the feeling that it's all a big rush to go somewhere." I think that's very true. Systemd was forced on the community, and it all happened very quickly. There was a lot of political maneuvering and strong-arming of distro maintainers to make it the default everywhere before it was mature.

Debian should not have adopted it for at least a few more years; they're supposed to be the slow, steady, and stable distro. Their quick move to systemd hurt a lot of feelings and caused half their team to leave for Devuan. That shouldn't happen. If your team is that fiercely split on an issue, the correct response it to leave the status quo alone until cooler heads prevail. Debian lost a lot of their reputation for stability because of this.

Honestly, I think if systemd had of been gradually adopted rather than forced, it would be a lot more popular. People could have seen Fedora and Arch use it for a few years and then maybe everyone would see the advantages and want to adopt it once it had proven stable. But the way it was, most of the community had no voice in the matter, and that's not a good thing.

I would have welcomed a sysVinit replacement that was more UNIXy. Something like systemd would have been nice, but systemd has too many flaws in both design and implementation to be what we needed.

I guess the silver lining is that it showed everyone the problems with the benevolent dictator model that Linux uses. I think people have been fleeing to the BSDs for the difference in community just as much as the technical superiority.

I find systemd's lack of faith in UNIX disturbing. So come join the BSD community. We have cookies ;)

published 2015-10-16