Then today, I found out Linux has a kernel module that allows you to "monitor" usb interfaces. Like... using wireshark.

I won't explain how to set up wireshark for your distro, that's for you to figure out.

Once you have wireshark set up, you just sudo modprobe usbmon then lsusb.

Find the USB interface you want to monitor in the list, then select it when starting wireshark (gui).

That's about it... you can filter like this:

usb.urb_status != 0 && !(usb.urb_status == -115)

Short post, but hope this helps you test your own USB cables for any errors that aren't shown in dmesg, like failed packets.

In this post I'll show you what a quadlet is and how you can use it to manage your containers.

Quadlets are a new way to manage containers using systemd. They are a new format for systemd units that allow you to manage containers using systemd.

Container Quadlets

Here's an example of a container quadlet.

[Container]
Image=docker.io/library/alpine:latest
Exec=sleep infinity
AutoUpdate=registry

If you put this in ~/.config/containers/systemd/alpine.container you can start it with systemctl --user start alpine and it will start a new container.

Just make sure to systemctl --user daemon-reload to reload the systemd daemon after you create the quadlet. You have to do this every time, I won't repeat this throughout the blog post.

Pod Quadlets

You can also use quadlets to manage pods. Here's an example of a pod quadlet.

[Pod]

Yep, that's it. If you put it in ~/.config/containers/systemd/mypod.pod you can start it with systemctl --user start mypod-pod and it will start a new pod.

But the best part is this. If you want to add a container to a pod all you have to do is this.

[Container]
Image=docker.io/library/alpine:latest
Exec=sleep infinity
AutoUpdate=registry
Pod=mypod.pod

Now when you start mypod-pod it will start the container as well.

Dependencies

You can add dependencies.

[Unit]
After=alpine.service

[Container]
Image=docker.io/library/archlinux:latest
Exec=sleep infinity
AutoUpdate=registry
Pod=mypod.pod

Now archlinux will start after alpine.

There are even more options. You can start .kube, .network, and .build quadlets too and they can depend on each other. See the docs for more info!

---

Example: Nginx and Certbot

Okay so now lets run something useful :) How about an nginx container and a certbot container to get a certificate for it?

Nginx Container

First lets create a directory to hold our files.

mkdir -p ~/.config/containers/systemd/www

Next we create a quadlet for our nginx container.

[Container]
Image=docker.io/nginx:latest
PublishPort=80:80
PublishPort=443:443
AutoUpdate=registry
Volume=./www:/usr/share/nginx/html

This will run a web server for us. If you aren't root, you'll need to enable unprivleged ports.

sudo sysctl net.ipv4.ip_unprivileged_port_start=0

Now we can start it with systemctl --user start nginx and it will start a new container.

If you browse to http://localhost you should see the nginx welcome page.

Certbot Container

Now we need to get a certificate for it. We can use certbot for this.

[Container]
Image=docker.io/certbot/certbot:latest
Volume=letsencrypt:/etc/letsencrypt
Volume=./www:/mnt
Exec=certonly --webroot --webroot-path /mnt --agree-tos --email your@email.com -d your.domain.com
AutoUpdate=registry

Make sure to replace your@email.com and your.domain.com with your own email and domain.

Then start the container with systemctl --user start certbot and it will get a certificate for you.

Nginx with SSL

Now you can use the certificate in your nginx container.

[Container]
Image=docker.io/nginx:latest
PublishPort=80:80
PublishPort=443:443
AutoUpdate=registry
Volume=letsencrypt:/etc/letsencrypt

That's great, but now we need to configure nginx to use the certificate. We can do this by adding a volume for the configuration file.

[Container]
Image=docker.io/nginx:latest
PublishPort=80:80
PublishPort=443:443
AutoUpdate=registry
Volume=letsencrypt:/etc/nginx/certs
Volume=./nginx.conf:/etc/nginx/conf.d

Along with a configuration file for nginx.

server {
    listen 443 ssl;
    server_name your.domain.com;

    ssl_certificate /etc/nginx/certs/live/your.domain.com/fullchain.pem;
    ssl_certificate_key /etc/nginx/certs/live/your.domain.com/privkey.pem;

    location / {
        root /usr/share/nginx/html;
        index index.html;
    }
}

Again, make sure to replace your.domain.com with your own domain.

Now you can start the container with systemctl --user start nginx and it will start a new container with the certificate. You can verify this by going to https://your.domain.com.

Now you have a web server with HTTPS.

Automatic Certificate Renewal

If you want to renew your certificate automatically it's as simple as this.

[Container]
Image=docker.io/certbot/certbot:latest
Exec=renew
AutoUpdate=registry
Volume=letsencrypt:/etc/letsencrypt

Now you can create a timer for it in ~/.config/systemd/user/certbot.timer.

[Unit]
Description=Renew Certbot certificates

[Timer]
OnCalendar=Sun 00:00:00
Persistent=true
Unit=certbot.service

[Install]
WantedBy=timers.target

Now you enable the timer.

systemctl --user enable --now certbot.timer

Now your certificate will be renewed every Sunday at midnight.

That's it! Now you have a web server with HTTPS and a certificate that will be renewed automatically.

Ostree is the software that allows rpm-ostree in Fedora Silverblue to keep snapshots of previous deployments and do incremental updates, but did you know it can also be used to keep snapshots of your own files? It's actually quite easy once it's all set up.

For this I would strongly suggest you use root for your ostree. Although you can use it without, you will be able to use files that regular users can't use such as hardlinks if you're root, which will speed up incremental snapshots very much.

Initializing the Ostree Repository

First you'll want to create your inital ostree. Do this in a new folder, on a drive with lots of free space for the files you want to snapshot. You'll need as much free space as all the files you want to snapshot.

mkdir tree
sudo ostree init --repo=tree

Committing and Restoring Files

Next you can start committing files to your new tree. Be sure to check the man pages for ostree-commit and ostree-init.

sudo ostree commit --repo=tree --branch=master $PWD/dir

Keep in mind you cannot commit individual files, only entire directories at a time. Now when you change something inside /dir you can commit again with the exact same command and you will have both copies in your ostree. To restore the original copy you can then use ostree-checkout.

ostree refs --repo=tree master

This will show you a list of all your commits. Now choose the one you want to restore.

ostree checkout --repo=tree --union your-commit-sha dir

Important Considerations

Make sure you check ostree -h for a list of ostree commands and read each of their man pages if you want to use them fully.

I use this on a filesystem that is already a RAID1. Make sure you keep proper backups, ostree can create snapshots but it won't help you if your entire drive fails.

If you've switched to Flatpak for gaming you may notice that Mesa is sometimes a little bit old. The issue I had myself was that my 6000 series AMD graphics card didn't have up to date graphics drivers, resulting in OpenGL games not running on the graphics card, but rather the CPUs OpenGL implementation, which made games such as CS:GO and Minecraft run at 5 or 10 FPS.

The Solution

To get the latest Mesa git on Flatpak you can install it like this:

flatpak install org.freedesktop.Platform.GL.mesa-git org.freedesktop.Platform.GL32.mesa-git

This will allow you to install the latest Mesa graphics drivers to allow applications to detect hardware properly and run better. Make sure when you are prompted for a version of Mesa to install you choose the latest version number that doesn't have "beta" at the end.

Usage

Once you've done this you can run your Flatpak like this to use Mesa git.

FLATPAK_GL_DRIVERS=mesa-git flatpak run com.valvesoftware.Steam

You can also export FLATPAK_GL_DRIVERS=mesa-git in your .bashrc or .profile to use Mesa git for all Flatpak applications. This may cause issues though, and I'd personally advise you only use it when it's needed.

I mentioned in my previous post that I use an external drive to keep my Steam games on. In an attempt to not have to re-download hundreds of gigabytes of games I keep them on a RAID1 with 2x5TB drives that I use BTRFS to manage. In this post I'm going to outline how I set it all up, and how I manage it when issues arise.

Formatting Drives

First of all I had to format the drives, so I made sure they were empty and everything was backed up. Don't do this unless you know for sure that all the data on all your drives is backed up somewhere else, or you want to delete it permanently.

Okay so the commands are easy, you just have to make sure you're using the right ones as not to lose your data. Don't run these commands without understanding what they do. Replace \_ with your drive letter. A lot of these commands need to be run with root, make sure to either elevate to root (sudo -s) or prepend the commands with sudo.

lsblk

mkfs.btrfs /dev/sd_1
mkfs.btrfs /dev/sd_1

Creating the RAID1 Array

Now you can mount the newly formatted drives to a single RAID1.

mount /dev/sd_1 /mnt
btrfs device add /dev/sd_1 /mnt
btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt

The last command might take a little bit of time, but once it's finished your drives will be all ready to be used.

Managing the Array

Now if you need to swap out a drive, add another, or you unplug one when it's writing you can simply balance again and btrfs should be able to balance back onto the drive.

btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt

You can view the status of the balance using:

btrfs balance status

You can remove a drive or add another using:

btrfs device remove /dev/sd_1
btrfs device add /dev/sd_1

Just make sure you balance right afterwards (which will take a VERY long time if you have lots of data, beware)

btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt

I was using Fedora Silverblue for nearly a year before I realized I had been using Flatpak's for nearly all my software, which mind you isn't much, but I can use Firefox, Chrome, and play any video game I want using Flatpak alone so it makes sense for me.

To get this working all you need is a distribution where you can install Flatpak. I'm using Alpine Linux edge with musl and even that works. Using Flatpak allows me to use proprietary video and audio codecs in Firefox and Chrome since musl doesn't support those codecs as of right now.

Setting up Flatpak

If your distribution doesn't already come with FlatHub like Alpine, you'll need to add it.

flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo

Installing Applications

This will allow you access to the entire Flatpak library. You can search for software using:

flatpak search steam

Then install using:

flatpak install com.valvesoftware.Steam

Make sure you periodically update your Flatpak's as well using:

flatpak update

Steam and Gaming

For Steam to work you'll likely need to install some other Flatpak's as well, but once they're installed all your other gaming Flatpak's should "just work".

flatpak install com.valvesoftware.Steam.CompatibilityTool.Proton

You might also want to install your distribution's steam-devices package if you want to use a game controller.

sudo apt-get install steam-devices # ubuntu / debian
doas apk add steam-devices         # alpine

Managing Permissions with Flatseal

I like to give the Steam Flatpak access to my external drives so I can keep games there instead of on my hard drive. To do this I simply install Flatseal and use Flatseal to give rw (read+write) access to the directory where my drives mount to.

flatpak install com.github.tchx84.Flatseal

Google Chrome

Next if you want to install Google Chrome.

flatpak install com.google.Chrome

Conclusion

That's all you have to do, then you would have Google Chrome installed.

Flatpak uses it's own runtime, remember to update all your Flatpak's periodically with flatpak update to avoid security vulnerabilities and other issues.

Prerequisites

To do this you will need:

1. make and gcc or clang
2. kernel-debug-devel package
3. sudo permission to insmod the compiled module

This[^1] is the only one I found that worked for me on the latest stable linux kernel, which as of today on Fedora is 5.17.6.

Kernel Header Installation

First you need to install the kernel development header package.

# Fedora
sudo dnf install kernel-debug-devel

# Ubuntu
sudo apt install linux-headers-$(uname -r)

[^2]

Building and Loading the Module

Next you need to make the module.

git clone https://github.com/morrownr/8814au.git
cd 8814au

Now simply make the module using the make command and use insmod to load it.

make -j16
sudo sh -c "modprobe cfg80211; insmod /home/codebam/8814au/8814au.ko"

If everything worked successfully, you should now have a loaded 8814au module that you can use along with your wireless card to connect to WiFi.

[^1]: https://github.com/morrownr/8814au

[^2]: https://www.tecmint.com/install-kernel-headers-in-ubuntu-and-debian

Podman[^1] has a nice feature called generate that lets you generate different kinds of configuration files for starting pods. One that I use often is podman generate systemd which generates a new user service that can be started with systemctl --user start container-yourcontainer.service or enabled.

Auto-Updating Containers

If you want your container to auto update you can simply pass it the flag --label "io.containers.autoupdate=registry" when you're creating it. The systemd service will automatically pull the latest versions of containers before starting them. You can manually update your container images now using podman auto-update.

If you want to update your images on a schedule you can enable the podman-auto-update.timer using `systemctl --user enable --now podman-auto-update.timer` which will auto update your containers every Monday morning.[^2]

Kubernetes YAML Generation

You can also generate a Kubernetes YAML configurations using `podman generate kube`. This can then be loaded in Kubernetes to run your containers.

[^1]: https://podman.io

[^2]: https://fedoramagazine.org/auto-updating-podman-containers-with-systemd/

Podman[^1] is a container management software similar to Docker that can run OCI containers as regular users (not root) by using container management libraries such as crun or runc, and cgroups v1 or v2. It can be used as a drop-in replacement for Docker and supports docker-compose using podman-compose.

Unfortunately for some distributions after installing podman the operating system defaults to cgroups v1 and runc, not cgroups v2 and crun. As was the case on Rocky Linux for me.

Switching to crun

First we can switch to crun instead of runc by installing crun with our package manager. It was already installed for me, but on Fedora you would use `sudo dnf install crun` Then you can edit the file /usr/share/containers/containers.conf and change the line runtime = "runc" to runtime = "crun". Now you can verify that you're using crun with podman info | grep crun.

name: crun
package: crun-0.20.1-1.module+el8.4.0+643+525e162a.x86_64
path: /usr/bin/crun
  crun version 0.20.1

Enabling cgroups v2

Now the harder part was enabling cgroups v2. Luckily I found a comment on a GitHub issue outlining the steps you need to complete to get cgroups v2 to work.

First we can enable it in the kernel with

sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=1"

Then we can enable linger. This will also allow our containers to remain running after logging out.

sudo loginctl enable-linger codebam

Then the last step which made everything work for me is to set the pid_limit=0. For me it was in /usr/share/containers/containers.conf instead of /etc/containers/.

Verification

Now you can reboot. podman info | grep cgroup should say

cgroupControllers: []
cgroupManager: systemd
cgroupVersion: v2

Now we're using cgroups v2 and crun.

[^1]: https://podman.io

Over the past few months I experimented with some new Linux distributions. Here I'll show you the one I'm currently using and why I think it's great.

Atomic Updates and Rollbacks

Fedora Silverblue is what's known as an Atomic operating system. On Fedora Silverblue every update is checked out similar to how git checks out a new reference. If you want to install packages into the base image you can use rpm-ostree install like you usually would, the only difference is that you need to reboot to boot into your new base system reference.

Toolbox for Development

If you don't want to reboot there is a better way though. Instead of layering packages into your base image you can use a tool called toolbox which is simply a wrapper around podman for creating containers with the current Fedora image.

You can use toolbox enter and install and run any software as well as access files inside your home directory.

Rebasing and Customization

If at some point you do updates and your system doesn't boot, you don't have to worry. You can simply reboot and choose the last working base image and your system will boot up as normal, allowing you to fix any breakage.

Silverblue is so much more powerful than that though. You can rebase your entire system on different base images or Fedora versions. Now instead of installing your new desktop environment and risk breaking your operating system you can do an rpm-ostree rebase and rebase on another desktop environment. I used this to rebase on an image from Kinoite called base which just includes all the base packages and no desktop environment. Then I installed my window manager of choice and it's dependencies. Now I have a fully modular Fedora installation along with Sway and all the dependencies I need to use it.

Alpine Linux has a cool feature called LBU that lets you create a backup on top of the Live ISO. These backups are called local backups.

I plan on using LBU as a system maintenance drive. It allows you to save changes you make to the Live ISO and restore them automatically when you boot the next time.

Setup and Committing

To set up LBU I booted the Alpine Linux ISO, ran setup-alpine, mounted my drive to /media/alpine_usb and when I got to the disk selection I selected no disk, and alpine_usb as where to store the configs, and the default for cache.

Now when I make changes in the Alpine Linux Live ISO I can commit them.

To see what is being committed I first check lbu status which lists all the files being added or removed, then lbu commit to commit the changes.

When I was rebooting I had an issue where the drive would automount itself to /media/sdb and then wouldn't work with lbu when I wanted to commit or check the status because it was already mounted. To fix this I just had to `umount /dev/sdb` on boot and then lbu worked as expected.

Caching and Configuration

To use cache you have to mount it to /media/alpine_usb so that it can write the cache files when you're installing packages. I want this because it makes reinstalling my packages on reboot a lot faster, they can just be loaded from cache instead of from online repositories.

Upon reboot I had to reinstall all the packages I had installed using `apk upgrade` but since the cache is there it just installs everything really quickly.

I uncommented and changed the number of backups in /etc/lbu/lbu.conf to 20 so that I would be able to revert up to 20 commits in case I broke my configs.

I see Alpine LBU as a good way to keep a backup of changes to the ISO so it could be used as a system recovery disk with the tools that are needed.

I'll start this post by saying one thing. Are you SURE you want to wipe that yubikey? I spent the last 2 weeks redoing my entire password manager. Silly me forgot that I had my passwords encrypted using the yubikey I wiped to create new PGP keys.

Anyways over the past week or two I moved all my passwords from pass into keepass and then back into pass when keepass didn't meet all my needs for a password manager. KeePass supported using yubikeys for decryption, but not more than one at a time. If I were to lose the key I would get permanently locked out.

Features I want in my password manager:

• 2FA for password manager decryption
• Mobile and desktop apps
• Secure encryption

What I settled on was this. I bought myself a [Yubikey 5C NFC](https://www.yubico.com/ca/product/yubikey-5c-nfc/) to go alongside my previous Yubikey 5C. There might be better alternatives, but this is the one I purchased. This one had NFC and was convenient for accessing passwords on my phone and laptop.

Yubikey PGP Key Generation

To generate my PGP keys I actually generated them on the Yubikey itself, this way the private key never touches the laptop I used to generate the key.

$ gpg --card-edit
gpg/card> admin
gpg/card> key-attr
gpg/card> generate

The key-attr command lets you choose the type and size of key you're generating on the device. I used RSA 4096 for my Yubikey 5C and ed25519 for the Yubikey 5C NFC. I can't use RSA or larger keys if I want to be able to use it with NFC.

I backed up my revokal certificates that were put in the ~/.gnupg/ directory as indicated after they were generated, and I exported and backed up the public keys, then I uploaded the keys to a keyserver.

gpg --export --armor 0F6D5021A87F92BA > 0F6D5021A87F92BA.asc
gpg --send-keys 0F6D5021A87F92BA

gpg --export --armor F9BC985B3BF972C7 > F9BC985B3BF972C7.asc
gpg --send-keys F9BC985B3BF972C7

I started putting passwords into pass using `pass init 0F6D5021A87F92BA F9BC985B3BF972C7`. That lets me encrypt to both Yubikeys at once. Now if I lose or accidentally wipe one of the Yubikeys again the other will be able to decrypt my passwords. I also signed the keys with each other, just to show that both are valid if one were to be lost.

While I did all this I thought I might as well enable 2FA using both the Yubikeys on every account that supported it. Google actually has something cool called Advanced Protection which allows you to set up your account with 2 yubikeys to login with, so I enabled that for my Google account.

I downloaded the [password store](https://play.google.com/store/apps/details?id=dev.msfjarvis.aps) Android app for my phone, along with Openkeychain.

This part wasn't too hard to set up, I just searched for my key on the keyserver that I just uploaded it to, then imported it. I went through the setup to add a security token and imported it so that it was recognized as one of my keys.

Git and Password Store Sync

At this point I set up my password store with git to use for syncing. I set it up on some mirrors just in case one went down, and I backed it up to Nextcloud.

git remote add all git@seanbehan.dev:pass
git remote set-url --add --push all git@seanbehan.dev:pass
git remote set-url --add --push all git@git.sr.ht:~codebam/pass
git remote set-url --add --push all git@git.maych.in:codebam/password-store
git push --set-upstream all master

I actually set up git on my own server here too. To do this I added a new user named git and added my ssh keys to it, then inside it's home directory I made git repositories to push to using.

mkdir pass
cd pass
git init --bare

Yubikey for SSH

I generated a new SSH key from within the password store app and used that for authentication from my phone. On my laptop I set up gpg-agent with ssh support so that I could use my yubikey as an SSH key as well as a PGP key.

~/.bashrc

export GPG_TTY="$(tty)"
export SSH_AUTH_SOCK="/run/user/$UID/gnupg/S.gpg-agent.ssh"
gpg-connect-agent updatestartuptty /bye > /dev/null

~/.gnupg/gpg.conf

use-agent

Then I used ssh-add -L to show my public SSH key for each yubikey to put on the server.

I used pass for 2FA as well because the password store app supports that. I just scanned the code and installed pass-otp on Fedora to get support for OTPs.

I installed the PassFF Firefox extention to get support for pass in my browser, and turned on autofill on my phone so that it can autofill everywhere.

I set up git to sign all my commits so that it shows that my commits are signed and valid.

git config --global user.signingkey 0F6D5021A87F92BA
git config --global commit.gpgsign true

System-level U2F Authentication

As an extra layer of security on Fedora I installed the pam-u2f package on and generated U2F logins.

sudo pamu2fcfg -uroot >> /etc/u2f_mappings

I ran it once for each key, then opened the file and edit it so it looked like this.

Before

root:key1root:key2

After

root:key1:key2
codebam:key1:key2

In /etc/pam.d/system-auth I added a line to the very top above all the other auth.

auth	    required              pam_u2f.so authfile=/etc/u2f_mappings

I added the same line to /etc/pam.d/su as well so that my user and root requires a yubikey to su.

This makes it so that sudo, su, and logging in through a getty (not in GDM for some reason, but I haven't fixed it) require you to tap the yubikey before typing the password. Two factors of authentication for my computer, and every online service that supports it, backed by hardware.

Once I was done all this I committed my changes to my dotfiles pushed them to git so that I could easily set the same thing up on my desktop which is also running Fedora.