Linux Certificate Authority root stores have a too simple view of 'trust' Let's start with the background. Pretty much every Linux system (really, every Unix system) has a 'system CA root store', by which we mean 'the list of all CA root certificates that are trusted by default by most TLS-using software'. For various sensible reasons, many Linux distributions reuse Mozilla's CA root store for the

Fedora 31 has decided to allow (and have) giant process IDs (PIDs) Every new process and thread on Linux gets a new PID (short for process ID). PIDs are normally assigned sequentially until they hit some maximum value and roll over. The traditional maximum PID value on Unixes has been some number related to a 16-bit integer, either signed or unsigned, and Linux is no exception; the kernel default

A perhaps surprising consequence of Requires dependencies in systemd I tweeted: Oh right. This service restarted because I said 'Requires=rsyslog.service' & I updated rsyslogd (which restarted it). Moral: don't do that. The systemd.unit manpage tells you that this will happen if you read it carefully. In the section on Requires: If this unit gets activated, the units listed here will be activated

It turns out that the meaning of 'load average' on Unixes is rather more divergent than I thought it was. So here's the story as I know it. In the beginning, by which I mean 3 BSD, the load average counted how many processes were runnable or in short term IO wait (in a decaying average). The BSD kernel computed this count periodically by walking over the process table; you can see this in for exam

My current views on using OpenSSH with CA-based host and user authentication Recent versions of OpenSSH have support for doing host and user authentication via a local CA. Instead of directly listing trusted public keys, you configure a CA and then trust anything signed by the CA. This is explained tersely primarily in the ssh-keygen manpage and at somewhat more length in articles like How to Hard

Today on Linux, ZFS is your only real choice for an advanced filesystem Yesterday I wrote about what I consider advanced filesystems are in general, namely filesystems with the minimum feature of checksums so you know when your data has been damaged and ideally with some ability to use redundancy to repair from damage. As far as I know, today on Linux there are only two filesystems that are advanc

I ranted about this on Twitter a few days ago when we discovered it the hard way but I want to write it down here and then cover why what Intel did is a terrible idea. The simple and short version is this: Intel switched one of their 'datacenter' SSDs from reporting 512 byte 'physical sectors' to reporting 4096 byte physical sectors in a firmware update. Specifically, we have the Intel DC S3500 80

Suppose, not entirely hypothetically, that you initially set up a server with one system disk but have come to wish that it had a mirrored pair of them. The server is in production and in-place migration to software RAID requires a downtime or two, so as a cheap 'in case of emergency' measure you stick in a second disk and then clone your current system disk to it with dd (remember to fsck the roo

System metrics need to be documented, not just to exist As a system administrator, I love systems that expose metrics (performance, health, status, whatever they are). But there's a big caveat to that, which is that metrics don't really exist until they're meaningfully documented. Sadly, documenting your metrics is much less common than simply exposing them, perhaps because it takes much more work

Let's start with the tweets: @thatcks: Everyone should strongly consider adding 'consoleblank=0' to the kernel command line on your Linux servers. #sysadmin @thatcks: The Linux kernel blanking the console screen is both unnecessary and dangerous on modern servers and modern setups. You want it off. By default if you leave a Linux machine sitting idle at a text console, the kernel will blank the di

Ssh-agent is probably normally used to handle a single identity key, but it can hold more than one if you want. One of the things that the ssh-agent and ssh manpages are a bit silent on is what happens if you have multiple SSH keys loaded into a single ssh-agent. Since I've been dealing with this as the result of transitioning from one set of SSH keys to another, I'm going to write down what I've

The bytes and events data for NFS mounts in /proc/self/mountstats The per NFS mount mountstats performance stats (see here for an introduction) have two sets of high level statistics, reported in the bytes: and events: lines. Both of these come from counters that are described in comments in include/linux/nfs_iostat.h in the kernel source. Of the two, the simpler is bytes:. A typical bytes: line l