The Source Will Be With You, Always

Thoughts and News

For all blog posts, see the archive.

FreeBSD and FreeNAS in Business


My father owns Charlotte Tent & Awning, a small business with about 30 employees, selling canvas and metal awnings to a wide variety of customers, from homeowners to massive shopping centers and sports stadiums. I'm going to tell you how FreeBSD and FreeNAS came to play a critical role in his business.


In the awning business, every awning must be custom-made to the dimensions of the building where it will be installed. There are also fabric and logo choices by customers that must be exactly right.

When contacted by a customer, the first step is for a salesman to visit their home, show fabric samples, and take measurements. If the customer decides to make the purchase, the work order is submitted by the salesman to the fabrication department, which welds together a frame, sews the fabric, and staples them together.

Once fabrication is complete, the installation department schedules a time to bring the awning to the customer's home and install it.

The whole process takes about four weeks, which means there are many active orders at any time. To complicate matters, there are sometimes modifications to an order partway through fabrication, times when they must wait for another company to ship them special fabric or parts, and special paperwork to handle for commercial or out-of-state jobs.

Keeping track of all ongoing jobs and making sure everyone has the most up to date information can quickly become a logistical nightmare. Any mistakes or miscommunications are very costly.

The Old Way

They used to manage most of this on paper. Photocopies of hand-written orders were distributed to everyone's physical mailbox, and any questions had to be run through the salesman responsible for the job. Outdated copies of orders were a problem, and the salesmen spent a lot of time coordinating.

Quickbooks handled billing. For photos of completed jobs, spreadsheets of all current jobs, and general status management, they used Dropbox and Microsoft Excel. This came with a number of problems:

  • There were frequent conflicts when someone forgot to save their changes.
  • Whenever someone added a photo to the company Dropbox account, it was uploaded to Dropbox, then downloaded 15 times in parallel to the other computers in the office, which caused bandwidth problems.
  • It was difficult to keep the information in Dropbox and on the printed work orders consistent.
  • Customer and job information had to be entered in multiple locations.

Complicating the internet situation was the fact that their consumer-grade router would reboot several times a day for no discernible reason.

Looking for Solutions

This is when my dad asked me for suggestions. He said that as they'd been growing, this had become less manageable. They needed something new.

At first, I looked at off-the-shelf solutions. I found a number of cloud services that were like Dropbox, but with some generic management stuff layered on top. Not only did these all feel like a poor solution, they were very expensive. If the provider were to go out of business, what would happen to my dad's company?

Being an engineer, I got frustrated with trying to put a square peg in a round hole and decided to write a system myself.

The Right Solution

I went through several iterations and had frequent meetings with the management staff, but we finally nailed down exactly what was needed.

I defined data models for customers, awnings, and jobs, and wrote a RESTful API with Nodejs, Express, and MongoDB (I now regret not using PostgreSQL, but that's a story for a different time).

I wrote the client-side app in Angularjs, with Bootstrap CSS and a mobile-first design. I plugged in Google OAuth so I didn't have to manage passwords, and gave each employee an account with varying permissions (e.g. an installer can't see any financial data).

The End Result

So what does it look like now?

  • All employees can view the status of all jobs, right from their phone, even when they're on the road; no more calling the salesman with every little question.
  • Every department can update a job with their status; no more situations where two people were both waiting on each other.
  • Job records are always up to date, because they're all stored in the same database.
  • Every job has a version number, so it's easy to follow the most recent information.
  • Customer information only has to be input once; no more handwriting.
  • Salesman can auto-generate PDF forms; no more misreading sloppy handwriting and no more time wasted photocopying handwritten notes.
  • The server does price and tax calculations automatically.
  • There's no dependence on some other company to stay in business.
  • Viewing the data on a phone integrates with Google Maps; no more printing out directions.

The Infrastructure

Fortunately, sourcing the hardware and setting up the OS was the easiest part; I talked to iXsystems. I ordered a FreeNAS Mini and a nice workstation tower, which were less than $4k, including hard drives and OS setup.

I had to fly to Charlotte to install the servers, so I scheduled a few days to make sure I could install everything and train the employees on the system I'd created.

The servers were waiting for me when I arrived, and I got to work. Unboxing stuff from iX is very nice. Everything was super padded, the hard drives were already screwed into the trays, and the trays were labelled. I literally just had to slide the drives in and power the servers up. The OS was already installed and the drives were ready with the ZFS configuration I requested.

I installed my database/web software on the tower server (r2d2, because it does the number crunching) and added shares for office personnel on the FreeNAS Mini (c3po, because of Samba shares and human interaction). And yes, I realize I named the tall skinny one r2d2 :P

I have r2d2 replicating ZFS snapshots to c3po, and the data is backed up off-site regularly. This data is absolutely mission-critical, so I can't take any risks. I'm glad I have ZFS on my side.

I replaced Dropbox with Samba on c3po, and the Windows machines in the office now store important data on the NAS, rather than their local drives.

I also replaced their router with an APU board running pfSense and replaced their PPTP VPN with OpenVPN and certificate authorization. Then upgraded the ancient 100 Mbit switch with a 1 Gbit model.

Because the server setup was already done for me, I finished with an entire day to spare.

Shortly thereafter, I recreated their company website with a mobile-friendly design, hosted in another jail on r2d2.

Looking Back

This setup has been in place for several months, and everyone is delighted with it. There are no more networking problems, fewer miscommunications, and much less time spent coordinating work. Efficiency is way up.

FreeBSD (in three different incarnations) helped me focus on improving the company's workflow without spending much time on the OS. And now there's an awning company that is, in a very real sense, powered by FreeBSD.

published 2015-12-02

Smartphones and Search Warrants


The Manhattan District Attorney's Office recently published a report on smartphone encryption and public safety. They argue that the best way to balance privacy and safety is to require companies like Apple and Google to be able to decrypt users' smartphones upon presentation of a valid warrant. The report is here

In episode 535 of Security Now, security expert Steve Gibson agreed with the proposal, saying that this is the right compromise. Leo Laporte disagreed. I was shocked to hear Steve endorse this, and I hope he'll reconsider. I'm writing this essay to explain why this proposal not only has legal problems, but is also fundamentally incompatible with both open source software and the principle that users own their devices.

Law Enforcement's Problem

To understand the issue, it's important to understand the problem that law enforcement agencies are trying to address -- finding digital evidence of crimes.

A warrant has traditionally provided law enforcement with the authority to go wherever they need to go and do (almost) whatever they need to do to find the evidence that they already have probable cause to believe exists.

From the perspective of law enforcement, the increasing role that digital records play in our lives and the discovery of methods that allow average citizens to easily apply strong encryption is problematic. It means there are digital spaces on computers and smartphones where a search warrant doesn't help them very much because they can't break the encryption in a reasonable time period (it may take thousands of years).

What law enforcement wants is for a warrant to enable them to search a computer or smartphone in the same way that they can search a room or a safe. It's a reasonable request, but I argue that this is neither legally nor technically feasible.

The Problem with Law Enforcement's Problem

Law enforcement has done an excellent job of convincing people that their ability to find criminals has been diminished due to the advent of modern encryption, and it's putting us all in danger. I argue that this is not the case.

Encryption is not New

Encryption is not a new discovery. We've been encrypting messages almost since we've had written language. The Caesar Cipher is a prominent example; Julius Caesar is known to have used it to encrypt military messages to protect their contents.

Not only is encryption ancient, it's a fundamental part of human expression. Children and adults alike use it regularly. We write words backwards. We substitute shapes or numbers for words or letters. We speak in Pig Latin for fun, and to prepare for bedtime without children understanding us. These are all examples of encryption.

When criminals keep incriminating written logs for financial transactions or times and places, they may use some form of hand-written encryption such that a casual observer would not realize the purpose of the writing. Criminals may talk about a "stack of pancakes" when they mean "box of bullets" to prevent others from overhearing.

My point here is that law enforcement has always had to deal with encryption. It is not at all a new discovery, as they would have us believe. Computers have simply made it easier to use.

Physical Places Still Exist

A warrant still authorizes them to search a home, bug a phone, or track a vehicle. Crimes always leave evidence, and only a small part of it may (or may not) be on an encrypted device. You're likely to find way more information in a person's home than on their smartphone.

If law enforcement cannot find sufficient evidence of a crime after following a suspect, searching his home and work, speaking with all the people he called, and viewing his financial records, then there's generally no reason to believe that unencrypting his smartphone will be any different.

Encryption Does not Prevent All Access

Look at how the FBI caught Ross William Ulbricht (known as the Dread Pirate Roberts, the founder of the drug-trafficking Silk Road), despite the fact that he used full-disk encryption on his laptop. They waited for him in the library. As soon as he sat down and decrypted his computer, the FBI jumped out and grabbed him. They got full access to his laptop, entirely bypassing the encryption.

There are still anomalies in how they identified him as a suspect, but the arrest is a perfect example of how law enforcement can legally arrest someone and bypass their encryption, without any need for a backdoor or key in escrow.

Cloud Providers

We live in a connected world, and the vast majority of our sensitive digital data is synced to cloud providers. Our email and contacts may be kept by Google. Our photos may be kept by Facebook. Our text messages may be kept by Verizon. Even our location is known to at least four different companies at all times. All these cloud provider companies are subject to court orders, without any need for special access to encrypted devices.

Making their Job Easier

At this point I have argued that encryption is not a new discovery, that in many cases law enforcement does not need any form of special access to bypass encryption, and that most of the data contained in an encrypted smartphone is also available elsewhere.

The real reason that law enforcement is fighting against encryption is simple: their job would be much easier if they could get around it. It's natural for anyone to wish their job were easy, but in the case of law enforcement, this is something we cannot allow.

What law enforcement really wants is to go fishing. That is, to collect data on a grand scale and have computers automatically sort through it to identify things that may indicate a crime.

In order to protect our God-given and constitutionally-acknowledged rights, the job of law enforcement must be a difficult one. In the United States, we hold that our rights are supremely important. While giving law enforcement special access to encrypted devices would catch a few more criminals, it would not be worth sacrificing our right to privacy.

We must remember that making law enforcement an easy job is the quickest path to tyranny.

This Particular Proposal

Apple and Google have both enabled full-disk encryption on iOS and Android phones as a default setting. Apple and Google do not have direct access to these devices any longer (only to the corresponding cloud accounts where data may be backed up).

Due to this state of affairs, the report from the Manhattan District Attorney's Office proposes federal legislation:

The federal legislation would provide in substance that any smartphone manufactured, leased, or sold in the U.S. must be able to be unlocked, or its data accessed, by the operating system designer.

There are a lot of problems, both technical and legal, why this won't work. I'm going to dig into all of them.

Legal Problems

Where would the authority for such a federal statute come from?

The Commerce Clause gives the federal government the authority to "regulate Commerce... among the several States," and "with foreign Nations." Because smartphones are part of interstate and foreign commerce, a federal statute regulating smartphones would comfortably fall within the power of Congress to regulate activities "that substantially affect interstate commerce."

Naturally, the Commerce Clause. I am not a lawyer, but I have studied constitutional law. This is typical logic for modern politicians: It is produced in one state and sold in multiple states, therefore the federal government can legislate anything about it we want!

It would be hilarious if it weren't such a common abuse. Interstate commerce is when an entity in one state trades with an entity in another state. It generally includes a semi-trailer truck crossing a state line with goods. Interstate commerce begins when goods are loaded onto the truck and ends when they are unloaded in the other state. The Commerce Clause gives the federal government the authority to regulate the manner by which goods are traded between people in one state and people in another. This means transport conditions, import tariffs, railroads and trucking lanes, etc.

It does not give them the authority to legislate how consumers use the goods or that the government have special access to them. Using the Commerce Clause to mandate special government access to consumer devices is clearly unconstitutional, given an educated reading of both the Commerce Clause and the Necessary and Proper Clause.

Technical Problems

The "operating system designer" would be required to decrypt the device and provide the data to law enforcement following a search warrant.

In the case of Apple and iOS, this is straightforward. iOS is a proprietary operating system written and owned by Apple, exclusively for hardware sold by Apple. Apple is clearly the "operating system designer."

What about Android? Android is open source. It uses the Linux kernel, which is under the GPLv2 license. The user space code is under the Apache 2.0 license. No one owns it; that's the whole point of open source. Aside from stock Android, there are many other versions; CyanogenMod is the most common alternative, but there are several dozen. So there is no "operating system designer" to which law enforcement could serve a court order.

Even carriers put their own version of Android on the phones they sell. But for the sake of discussion, let's assume that Google is somehow responsible for all Android variants.

How Encryption Works

To see the implications of this proposed legislation, it's important to understand how full-disk encryption is implemented.

Full-disk encryption works by translating all data saved to or read from a drive. When you save data, the encryption layer first encrypts it and then writes it. When you read data back, it reads the encrypted data, decrypts it, and then gives you the decrypted version.

As with any method of encryption, you need a key to input into the encryption algorithm in addition to the data. The key is subject to the following properties:

  • If the key is lost, the encrypted data can no longer be recovered.
  • If the key is weak, the encrypted data could be accessed by guessing the key.
  • If the key were to change, all the encrypted data would need to be decrypted with the old key and then re-encrypted with the new key, which is not feasible.

Full-disk encryption is generally accessed by providing a password, although it could also be something like a fingerprint. Passwords are subject to the following properties:

  • They are weak (easily guessable, not random).
  • They must be changeable.

Looking at these two sets of properties, it's clear that encrypting a drive with a password is not ideal. To address this, full-disk encryption systems use the following system:

  • A strong symmetric key is generated.
  • All saved data will be encrypted with that symmetric key.
  • The key is encrypted with the user's password.
  • The encrypted version of the key is stored in a special location on the drive.

I've simplified it a bit, but those are the key points. In actuality, you'd use a key derivation function like PBKDF2 to turn the password into something stronger before using it.

To turn on the device, the following steps take place:

  • The user provides their password.
  • The key is read from the drive and decrypted.
  • The decrypted key is remembered as long as the device is on.
  • When any data is read, the key is used to decrypt it on the fly.

This is why the data is only protected when the device is powered off; so long as it's on, the decrypted key is still in memory and anyone holding the phone can access everything (assuming the screen is unlocked).

Special Access

So how would this key escrow proposal work? To comply with the proposed law, Apple and Google would need access to the key used to decrypt the phone's drive. The key must not be the same for each phone, or else my decrypted key would also open everyone else's phone. So rather than storing one copy of the encrypted key, you store two:

  • A strong symmetric key is generated.
  • All saved data will be encrypted with that symmetric key.
  • The key is encrypted with the user's password.
  • The key is separately encrypted with Apple's public key.
  • Both encrypted keys are stored in a special location on the drive.

In this system, a user still decrypts their phone like normal. The phone's data is still protected against theft. But now Apple can use their private key to access the device too, much like a user using a password.

At first glance, this seems like a good compromise. Our devices are very well protected; A stolen phone would still be reasonably secure. Only Apple or Google could access it, given both physical access to the phone and a court order.

The Problems

But there's a big problem here. What is stopping me from removing Google's version of the key? What if I write a bunch of zeros to that section of the drive, erasing their copy of the key? Then I would once again be the only one who could decrypt the phone.

To comply with the law, Google would have to prevent me from doing that. But how? The phone is mine, and I have control over it. They can't stop me from erasing their copy of the key.

The only solution to this problem is for them to take away my control. They'd have to restrict what I can do with it, and that means restricting what software I can run on it.

Apple could actually do this if they wished, because they already don't allow the users any control. But Android's open source license prevents Google from doing this. Any code they add to generate and protect their copy of the key could be removed by an average user -- you only have to flash it with a different version of Android that wouldn't include this code, like CyanogenMod.

Android phones would then have to prevent you from modifying the operating system. This practice is known as tivoization. It uses hardware restrictions to prevent open source software from functioning in an open manner.

It's obvious at this point that any such proposal is fundamentally incompatible with open source. It requires proprietary operating systems and locked-down devices. It removes the power of choice from consumers.

We've already seen what happens when consumers don't control their own devices. I was forced to retire my previous Android phone because my carrier wasn't releasing Stagefright patches and the locked bootloader wouldn't let me install a different version of Android. Carrier control over devices is a huge security problem, and it's the biggest challenge facing Android today.

With cell phone unlocking being legal and the FCC encouraging wireless providers to unlock phones, a law that effectively requires all phones and tablets to be locked down is a clear step backward.

And contrary to what the report asserts, this proposal would impose a significant burden on "operating system designers," who would not only need to maintain an additional public key infrastructure, but also implement complex code to attempt to prevent deletion of their key.

What about Computers?

This proposal is specifically about smartphones and tablets, but there is little or no difference between a tablet and a laptop these days, with many devices that claim to be both. What if we apply this to all computers?

The same conclusion applies here. It would be illegal for hardware manufacturers or "operating system designers" to sell a computer that could be encrypted in such a way that the "operating system designer" could not recover the data.

This gets absurd quickly. Not only would we be unable to install Linux, BSD, or Illumos, Plan 9, or any of the great variety of open source operating systems, there's a big problem with removable drives. What prevents a criminal from attaching an external USB drive and storing encrypted data there?

Encryption is Math

There is a fundamental problem with any attempt to grant law enforcement special access to encrypted data, which is that encryption is just math. It's not a product or a service or a physical thing you can control. It wasn't invented; it was discovered. It's really just math. You can no more legislate special access to encryption than you can legislate the value of pi.

This is why any attempt to give law enforcement special access to encrypted data is doomed to failure. Even if this proposal were made law, it would be trivial to create an app that stores one's own encrypted data to the drive. Law enforcement could decrypt the drive, but all they would find is more encrypted data.

If they ban all apps that do encryption, then we lose web browsers, as well as any app that connects to 'the cloud'. Even then, users could sideload their own encryption apps, bypassing the app store.

The short answer is that it's not possible to do what law enforcement has requested. No matter what scheme you come up with, it can be easily bypassed, even by a user with little technical knowledge (as with rooting or jailbreaking a phone). It would be just as anti-consumer and just as ineffective as DRM.

Default Settings

So what if, rather than mandating that "operating system designers" must be able to access the data, we reduce this to a best-effort approach? They must be able to access data from a device using default settings, but are not held accountable if the user has manually removed their keys or bypassed their access mechanism.

This removes most of the issues I've raised about the proposal, but it also removes any benefit that special access would provide to law enforcement. Anyone who felt that protecting their privacy from government access was worth the extra effort would remove the second key. It would be very similar to rooting or jailbreaking a phone.

In the end, the criminals that the government is targeting would be immune to law enforcement's special access, which would make this proposal pretty pointless.


Once again, what law enforcement has asked for just isn't possible. No amount of beating up Silicon Valley nerds is going to make it any less so.

The right answer is for law enforcement to rely on traditional methods, which have only gotten easier to employ. If they need access to an encrypted device, they can arrest the suspect while the device is decrypted. Even without the devices themselves, they still have access to our most important data, which lives with cloud providers. And they have more metadata than they could ever hope to sort through.

They already have their golden age of surveillance. It's time they stopped asking for more.

published 2015-11-30

Problems with Systemd and Why I like BSD Init


In my first post on this blog, I said that systemd was one of the things that pushed me to look into the BSDs and eventually move everything to FreeBSD. People have asked me what my problem with systemd is, but I haven't written about it. So finally, here are my thoughts.

I've been reluctant to discuss systemd there because I was sick of everyone fighting over it. The Gentoo mailing list (and everyone else on the internet) became a giant flame war where everyone either thinks that systemd is evil and attacks Lennart Poettering personally, or thinks that systemd is going to save us all from ourselves and everyone must use it. I was just sick of everyone being so unreasonable.

For my part, I'm not a fan of systemd but I also don't think it's the end of the world. I watched a great interview with Lennart on the Linux Action Show about why he implemented it, and he had some good reasons. To write a daemon for Linux, you need to maintain a different init script for each distro because they all put things in different places. And sysVinit isn't the best with dependencies. Developing things for the Linux desktop is not as easy as it could be, due in large part to the fragmentation.

I avoided it when I was running Linux at home, but these days I have to use Linux for work and can't avoid it any longer. After using systemd for a while, there are definitely some things I like about it. It makes easy things easier. Like taking some daemon I wrote and making it a service. The service files are nice, and telling it keep restarting my service if it crashes is super handy. I like that the service file will work on any Linux distro with systemd. I can just about write a service file from memory, whereas with an rc script I'll always need to start from an example.

On the other hand, it makes the hard things even harder. If I want to do something complex that systemd doesn't have an option for, I'm out of luck. With BSD init, I can just edit the /etc/rc script and replace it with arbitrary shell code. By default it looks under /etc/rc.d/ and runs the enabled startup scripts there, but I can change that in less than five minutes.

Here's FreeBSD's /etc/rc. /sbin/init calls this script, and /etc/rc calls all the enabled scripts under /etc/rc.d/.


Notice how tiny it is. I can change anything in here I want. I can remove the part where it starts services and manually start just one or two services. I can change where the console output goes, like redirecting it to a file. I can make this script call whatever I like, however I like. Unlimited flexibility.

I'm not an expert in init systems, but I used OpenRC for a long time and I'm pretty familiar with BSD init now. From what I understand, BSD init is superior to the old sysVinit that Linux used to use. And Gentoo's OpenRC is similar to BSD init. I don't have much experience with classic sysVinit. So that's where I'm coming from.

Systemd is very opaque. What I mean by that is that I can't easily debug it. I can't look inside and see what it's doing, because it's a complicated binary. I can't easily make changes to the way it behaves. BSD init is just a script, so I can look at it, add print statements, and change it in any way I like. It's easy to look at it and understand exactly what it's going to do and why.

The way systemd does parallel startup bothers me. If service A depends on service B, they were traditionally started in sequence; A only starts after B is done starting. Systemd will start them both at the same time, but buffer any messages A tries to send to B until B is ready. So A just thinks that B is being slow to respond, and system startup is faster. You can disable this for certain things by using the 'After' directive, but that doesn't apply to the whole system and anyhow, disabling a feature isn't an answer to whether it should exist in the first place.

My problem with this is that the order in which services are started should, in my opinion, be exactly the same each time and predictable to the sysadmin. With systemd, the order is not deterministic, so you don't know what's going to happen next time you boot. I work with servers and embedded devices; I don't care much about boot time. A server spends several minutes in the BIOS during POST anyway, before the bootloader is even run; making the OS boot faster doesn't change very much. Embedded devices already start quickly because you trim them down to the bare minimum. What I care about is that every time I boot, the same exact things happen in the same exact order -- the order that I want.

It seems no one can agree on whether systemd is modular or not. I think the problem is with different definitions of 'modularity'. Systemd doesn't put everything in PID 1 like some people suggest; it uses modules that communicate with each other. So it is modular in that sense. But these modules are very tightly integrated. You can't easy remove some of them, or replace them with other things. So in that sense it is very monolithic. This is not at all like having a simple interface and passing data via stdin and stdout, which is the modularity that makes UNIX pipes possible. This is the sense that matters to me.

I dislike the way systemd is absorbing everything. It's not just an init system, it's become an everything-under-the-hood includes-the-kitchen-sink management system. That doesn't feel modular to me. Why should systemd implement NTP when ntpd already exists? I think systemd-timesyncd and all the others like it are just reinventing the wheel.

In the case of ntpd, you might say that it's a monstrosity that needs to be replaced with something much smaller, and you'd be right. There are others re-implementing it. OpenBSD created OpenNTPD, which is very lightweight, client-only, and can even hit HTTPS sites to verify that the times it gets from NTP servers is reasonable. Cleaning up or re-implementing ntpd is a good thing, but OpenBSD didn't feel the need to make it part of init.

EDIT 2015-10-31: Thanks to @myfreeweb for correcting me that OpenNTPD is not client-only. It is, however, client-oriented, as it sacrifices some accuracy for simplicity. If you're running a time server, you'll probably be using something else.

Linux is very fragmented and systemd is bringing things back together a bit. That may be a good thing. But the way we're headed, there will come a point when we won't be able to say 'GNU/Linux' anymore. It'll be 'GNU/systemd-linuxd'.

But part of what has made Linux so special is the ability to swap out literally any component on the system with something else -- even libc! All a Linux program needs is a kernel and something that looks like a libstdc.so. It's common to build embedded Linux devices with ulibc, or something else tiny. BSD can't just replace its libc, it's too integrated.

There was a really good interview with some GNOME devs on BSD Now not long ago. They said they were going to pay more attention to BSD compatibility, but also talked about why they needed to depend on systemd: There is no API or standard way to do things like set the timezone, change wifi networks, or perform many common desktop tasks on Linux. GNOME's code is full of #ifdefs so that they can use different code for each distro or OS, and it's not maintainable. Systemd provides these APIs in a clean and cross-platform manner, which is a big win for desktop environments.

OpenBSD is working on systemBSD, which is a systemd shim. It provides the APIs that GNOME needs, but doesn't implement the rest of systemd. I think having APIs like this is a really good thing, and I'd like to see us as an industry move toward standardizing on it (or something like it).

Systemd still feels unstable. I'm not sure whether it's because it actually has more problems than other software or just because people are quick to publicize its flaws, but it's had some problems that would be really funny if they weren't so serious. I've included links to the actual bug reports:

For a critical piece of infrastructure, having regular bugs like this is a big problem. It renders machines unbootable and ruins people's days. Compare this to a really well-managed project like OpenZFS. To my knowledge, no one has ever lost data as a result of a bug in ZFS. Infrastructure projects like this have to be held to a higher standard than most software because the consequences for a bug can be so severe.

I think the systemd devs suffer from NIH syndrome. I can't find anything to suggest that they considered OpenRC; they seem to have a personal problem with Gentoo. Many of the problems they described with init systems are already solved by both OpenRC and BSD init. Now they're going around re-implementing every possible Linux daemon as a systemd module, just because they can.

I was talking to a reader recently who said, "Every time I look at systemd development, I get the feeling that it's all a big rush to go somewhere." I think that's very true. Systemd was forced on the community, and it all happened very quickly. There was a lot of political maneuvering and strong-arming of distro maintainers to make it the default everywhere before it was mature.

Debian should not have adopted it for at least a few more years; they're supposed to be the slow, steady, and stable distro. Their quick move to systemd hurt a lot of feelings and caused half their team to leave for Devuan. That shouldn't happen. If your team is that fiercely split on an issue, the correct response it to leave the status quo alone until cooler heads prevail. Debian lost a lot of their reputation for stability because of this.

Honestly, I think if systemd had of been gradually adopted rather than forced, it would be a lot more popular. People could have seen Fedora and Arch use it for a few years and then maybe everyone would see the advantages and want to adopt it once it had proven stable. But the way it was, most of the community had no voice in the matter, and that's not a good thing.

I would have welcomed a sysVinit replacement that was more UNIXy. Something like systemd would have been nice, but systemd has too many flaws in both design and implementation to be what we needed.

I guess the silver lining is that it showed everyone the problems with the benevolent dictator model that Linux uses. I think people have been fleeing to the BSDs for the difference in community just as much as the technical superiority.

I find systemd's lack of faith in UNIX disturbing. So come join the BSD community. We have cookies ;)

published 2015-10-16

Following Tech Industry News


Keeping up to date with industry news is very important, especially in the fields of technology and security, where things are moving so quickly. I'm pretty happy with my current system, so I thought I should write about it.


Podcasts are my primary means of news. I love them because I can listen to them in the background when I'm working on simple tasks or cleaning the house or whatever.

Security Now

Security Now is an amazing show. Steve Gibson has a talent for explaining technological concepts in easy-to-understand ways. He doesn't just report news, he takes a deep dive into exactly what happened and why it's a problem.

Leo Laporte is a great counterpart. He and Steve talk like old friends, which I suppose they are after doing the show for so may years. They disagree on some issues relating to advertising, which makes for a very educational dialogue.

This Week in Enterprise Tech

Fr. Robert Ballecer (The Digital Jesuit) hosts this show about all sorts of news relating to the enterprise. He's an amazing host because he knows how to keep a discussion moving quickly, and always asks good leading questions for his guests to answer. They talk about data center and storage technology, BYOD policies, and general news.

This Week in Law

Hosted by Denise Howell and Sarah Pearson, this is unlike the other shows I watch. These people are lawyers specializing in areas like copyright, patents, and general application of technology. They talk about Bitcoin, space, software, etc. A good bit of it is over my head, but I feel like I'm learning a lot.


BSD Now is hosted by Allan Jude and Kris Moore. Until Recently, TJ was the writer/producer. This is all about the BSDs, with a natural emphasis on FreeBSD. They cover news and (most importantly) interview developers and people who use BSD in interesting ways. It's a fantastic show, and it guided me through switching from Linux to FreeBSD. It's rare that you get to hear directly from the developers of the software you use.


TechSNAP is hosted by Allan Jude and Chris Fisher. They cover news and security events relevant for sysadmins. It's similar to Security Now, but from a sysadmin perspective rather than a security researcher perspective. I find both valuable.


In addition to the regular shows I listen to, I also watch a lot of recorded talks on youtube.

Google I/O

Google I/O talks are of very good quality. The presenters are generally very good at presenting, the news is relevant to my field, and they have interesting things to say. I learned most of what I know about mobile web development from Google I/O talks.

BSD Conferences

The BSD conferences sometimes record their talks, and they're fantastic. BSDCan, vBSDcon, and even the local user group meetings. You can hear explanations of new code or features, as well as design discussions between developers before things get committed.


I'm a latecomer to twitter because I've been avoiding social media. But I gave it a try and now twitter has replaced my RSS reader as a source of news articles. I follow particular researchers and journalists, like Steve Gibson and Brian Krebs. I can also catch some of the conversations between various BSD developers and people in the community.


Usenet is worth a mention. I don't spend as much time there as I used to, but it's a great medium. Purely text, not under anyone's control, and there are lots of small communities there. It's a good place for discussions. Head to Eternal September if you want to get started.


This is a lot to keep up with, but I listen to podcasts and the like while doing something else, and only follow a small number of people on twitter. For me, this is as efficient as I can be in an effort to stay informed without wasting time on it. If anyone knows of similar things I might be interested in, feel free to drop me a line.

published 2015-09-16

Reporting Bugs and the BSD Community


Last time I wrote about Moving to FreeBSD. Now that I've been using it for a while, I have more to say.

There was a time when I didn't report bugs. I was lazy. I didn't want to invest the time trying to diagnose the problem, so I worked around it. I would use a different program. I would bump whatever I was doing to the bottom of my TODO list and try building it again in a week or two. It would get fixed eventually. Probably.

That wasn't very good of me.

I've been using Linux for the past several years and BSD for the past few months. I love the open source community. I've benefited immensely from this wealth of knowledge, code, and cooperation. I write lots of code including C, but not for any major open-source tools. I'd never contributed to an operating system. But I wanted to give something back.

I spent time brainstorming ways in which I could contribute back to the community. I made lists of all the tools and programs that I use and rely on. But none of them jumped out as things I would enjoy working on.

So I tried to think of new programs that were needed. But that never went anywhere. I never had that much time, and couldn't think of anything that was needed anyway.

I didn't want to keep being such a freeloader, but didn't know what to do about it. I don't think this is unique to me, which is why I'm writing about it. There are threads on slashdot about it and websites dedicated to connecting coders with projects. But many times, figuring out how to contribute is not so easy.

When I started using FreeBSD, I appreciated the greater emphasis on documentation, correctness, and proper design. It was refreshing. My operating system felt more professional, and I wanted to be part of the community in a way that Linux never pulled me in.

The FreeBSD documentation has a great article on how to contribute. So I decided that I was going to start reporting bugs. I read about how to file a PR. I was going to spend a little extra time investigating problems I came across, and I could both get my problem fixed and make FreeBSD better. What a great deal!

One of my many hats is as an embedded systems engineer. Any time you use an OS on an architecture that isn't mainstream, you're going to run into more problems than normal. Maybe someone enables a particular compiler flag, but has no way to test it on ARM. Even if they did, it's a time-consuming process that requires hardware and a build environment.

To date, I've filed seven PRs for FreeBSD, most of which relate to ARM. Three are fixed, three are just waiting to be committed or for someone else to respond, and one is a kernel issue that probably won't be fixed until I can figure out how to debug and fix it myself. One of my standing goals is to learn enough to resolve all the PRs I've opened.

One PR was fixed and committed in less than 20 minutes! My productivity was barely affected. Sometimes it's faster to report the bug and get it fixed than it is to work around it.

The next step for me is to start submitting patches with my PRs. I've been reading the Porter's Handbook, and these makefiles are starting to make sense to me. The differences between BSD make and GNU make still throw me off sometimes.

I've been reading other books as well. I read FreeBSD Mastery: Storage Essentials by Michael W Lucas and I'm almost done with FreeBSD Mastery: ZFS by Michael W Lucas and Allan Jude. Both wonderful books that save you many days of scouring man pages and Google.

But the big one I'm reading is The Design and Implementation of the FreeBSD Operating System by Marshall Kirk McKusick, George V Neville-Neil, and Robert N.M. Watson. It's a textbook that focuses on the kernel. It's dense material and it's going to take me a while to get through it, but the information is presented well. Operating system design has been interesting to me recently, and I like how the book walks me through all the design decisions in the kernel, and the history of how it happened.

I'm learning all this because it's fun and interesting, but also because I want to be part of the community. I'd like to earn my way to being a committer. I feel this way because the BSD community is friendly, professional, and knowledgeable. It's a meritocracy. Everything the Linux community isn't, but should be.

So please, report bugs. No matter what operating system you use. Even if you're not sure how. Don't be a freeloader. Participate in the software you use, and make it better.

published 2015-07-07