Jonathan E. Magen / @yonkeltron

PLUG: Philly Linux User Group

What is this?

A talk presented to the Philly Linux User Group (PLUG) on January 6th, 2021. The topic is "Linux Systems Programming with Rust" and focuses on Linux-specific programming with the Rust Programming Language. This talk will feature exactly zero slides and will be made entirely available, complete with working example code, online.

What do you need to know coming into this?

  • How to program or read programs.

  • What it is, the Linux.

Do you need to know Rust?

  • No.

Who is this Jonathan guy?

  • Principal Computer Scientist

  • 6 years at current company (healthcare)

  • 5 years in startups before that

  • Linux user since the early days

    • Early days means

      • Prior to RedHat Linux 9!

      • Started when Linux Live CDs were very new

  • Member of the PLUG mailing list for well over 15 years

Tonight’s Tech Stack

Equipment

  • Linux running inside a Google Pixelbook Chromebook!

Software

Why are we here?

To learn about Rust and Linux!

Agendanomics

  • Some definitions and history

  • Intro to Rust

  • Linux systems programming

  • Examine a real application: Hermione

    • Q&A with co-maintainers

  • Future work

What is Systems Programming?

Systems Programming

Broadly

Non-app programming like:

  • OS development

    • Kernels

    • Drivers

  • System software

    • Daemons

    • Infrastructure components

  • Frameworks and libraries

    • Game engines

    • Windowing toolkits

This term is a bit silly.

Why is this definition silly?

At face value you could do "Systems Programming" with:

We typically think of systems programming languages as natively compiled ones.

Whatever.

Traditionally, systems programming has been the domain of C and C++.

  • Especially in the Unix world

    • GNOME desktop software written with GTK in C

    • KDE desktop environment written with Qt in C++

  • Certainly on Linux

Mistakes were done.

Unchecked, manual memory management leads to bugs!

So now Microsoft and Google are in agreement?

Enter the challengers!

We’re obviously here for Rust, though.

Some active Rust OS dev projects

  • Redox is a Unix-like microkernel OS

  • Tock is an OS for IoT

  • Firecracker is an AWS-sponsored project for VM, container, and function-based services

What about ?

Rust Linux

Rust Windows

Rust macOS

And I all of you!

Ok, sure.

No, seriously. Rust is fantastic!

  • Originally developed at Mozilla

  • Used by many, including AWS

  • Designed with some very novel features

Novel features of Rust

  • Safety

  • Ergonomics

  • Efficiency

Safety baked into types

Affine types

  • From affine logic, a substructural logic

  • Values may be used at most once

If this sounds weird, it’s because it is. Weirdly wonderful.

Safety enforced by the compiler

Borrow checker

  • Makes sure your code doesn’t use values it shouldn’t

  • Higher learning curve

Added to D, being added to Swift.

Evern more of Rust’s safety mechanisms

  • Compile-time memory management with lifetimes

    • Compiler does the hard work for you

    • Fine-grained control, without malloc and free details.

  • No null or equivalent, Option<T> instead

Ecosystem ergonomics

  • Best compiler I’ve ever worked with

    • Fantastic error messages

    • A bit slow, though

  • Great tooling

Rust-the-language cares about users

Incredible linguistic attention to programmer productivity:

  • Functional programming constructs come standard

  • Pattern matching

  • Expressions

  • Macros

  • Objects (structs) but no inheritance

    • Traits!

Traits are a bit different from Scala’s implementation. This remains mostly due to their deliberate simplicity and an equally deliberate omission of Higher-Kinded Types (HKTs).

Less terrible error handling

Compiler-checked errors with Result<T, E> to mark fallible computation

  • No exceptions!

  • Single return values

  • Error propagation made simpler

Efficiency

  • Zero-cost abstractions

    • You don’t pay for what you don’t use

  • Optimizing compiler

    • Slow because it does a LOT!

  • Speed, relative to C: ~90%

Concurrency and parallelism

Rust on the web

Sounds good.

Yes. It is pretty good.

It is not, however, perfect.

  • No map-literal syntaxes

    • There are macros, however

  • High guardrails sometimes complicate simple tasks

  • Slow compilation times elongate the "inner development loop"

  • Ecosystem still growing

    • Several parts are still immature

Back to !

So where does Linux come in

Several places

  • Linux software being written in Rust

  • Linux-specific libraries for Rust

Lots of Linux software being written in Rust

  • vopono Manage per-app VPN tunnels

  • kmon Linux kernel monitor + activity

  • lfs Linux filesystem info tool

Helpful Rust crates (libraries) for systems programming

Some of my favorites:

  • libc - Foreign-Function Interface (FFI)

  • nix - Friendlier *nix bindings

  • procfs - Interface to /proc

  • caps - Linux capabilities

This talk will become more about Linux-specific programming

Let’s look at some code!

Goals

  • Maintain realism by using actual libraries.

  • Show how to use Linux-specific functionality where possible.

  • Explain examples with context.

Non-goals

  • Exhaustive introduction to Rust

  • Cross-platform code

  • Exhaustive environmental overview

We will first build a Linux process viewer!

First thing’s first:

You can install Rust with rustup

Next:

Meet cargo!

(Cargo is Rust’s build tool.)

Crates we will use

Add our dependencies to the Cargo.toml file

[dependencies]
color-eyre = "0.5"
paris = "1.5"
procfs = "0.9"

Add code to our project

At a high-level:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

(1)
use color_eyre::eyre::Result;
use paris::Logger;

(2)
pub fn view_procs() -> Result<()> {
  let mut logger = Logger::new();

  logger.info("Starting up!").newline(1).log("Processes:");

  (3)
  procfs::process::all_processes()?
    .into_iter()
    .map(|process| {
      format!(
        "{}: {} - {} bytes",
        process.pid, process.stat.comm, process.stat.vsize
      )
    })
    .for_each(|process_message| {
      logger.indent(1).info(process_message);
    });

  Ok(())
}
1 Preamble
2 Function definition
3 Main meat of the program

Let’s break this down!

Preamble and first bits

(1)
use color_eyre::eyre::Result; (2)
use paris::Logger; (3)

(4)
pub fn view_procs() -> Result<()> {
1 Imports
2 Colored error handling
3 Stylish logging output on the console
4 The primary function is fallible and so returns a Result

Logging some output

  (1)
  let mut logger = Logger::new();
  (2)
  logger.info("Starting up!").newline(1).log("Processes:");
1 New up a logger, which is marked as mutable with mut
2 Emit some friendly output to the terminal

Remember:

The ? operator either returns the contents of the Result or short circuits by bubbling up the error to the calling function!

The guts of the process viewer

 procfs::process::all_processes()? (1)
    .into_iter() (2)
    .map(|process| { (3)
      format!(
        "{}: {} - {} bytes", (4)
        process.pid, process.stat.comm, process.stat.vsize
      )
    }) (5)
    .for_each(|process_message| {
      logger.indent(1).info(process_message);
    });
1 Query all processes from /proc
2 Get them in an iterator
3 Map processes to `String`s
4 Grab the PID, name, and memory usage
5 Log each string!

Close it out, bring it home

  (1)
  Ok(()) (2)
}
1 Signal that it all went well by returning an empty Ok
2 Note: no semicolon means a return expression

Walla! We’re done!

Less than 25 lines, with spaces!

It doesn’t have to feel low-level to be low-level.

Rust usually feels high-level.

Ok. Now what?

Next, let’s explore the wide world of filesystem event notifications provided by inotify!

inotify(7) is money, but confusing!

The nix crate makes it much simpler, though!

Let’s write a little inotify program which watches for filesystem changes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

(1)
use color_eyre::eyre::Result;
use nix::sys::inotify;
use paris::Logger;

(2)
pub fn setup_watcher(path_str: &str) -> Result<bool> {
  (3)
  let watcher = inotify::Inotify::init(inotify::InitFlags::empty())?;
  let watch = watcher.add_watch(path_str, inotify::AddWatchFlags::IN_ALL_EVENTS)?;

  let mut logger = Logger::new();
  let mut go = true;

  (4)
  while go {
    logger.newline(1).loading("Waiting for events...");
    let events = watcher.read_events()?;
    logger.info(format!("Got {} events", events.len()));

    for event in events {
      let msg = format!("Event: {:?} for {:?}", event.mask, event.name);
      logger.indent(1).log(msg);
    }
  }
  (5)
  watcher.rm_watch(watch)?;

  Ok(go)
}
1 Preamble
2 Function definition
3 Setup
4 Main logic
5 Clean up

Again, we’ll break this down!

(1)
pub fn setup_watcher(path_str: &str) -> Result<bool> {
  (2)
  let watcher = inotify::Inotify::init(inotify::InitFlags::empty())?;
  (3)
  let watch = watcher.add_watch(path_str, inotify::AddWatchFlags::IN_ALL_EVENTS)?;
1 Create our function which takes a path as a string slice
2 Initialize our watcher
3 Create the watch!

Setup for main loop

  (1)
  let mut logger = Logger::new();
  let mut go = true;

  (2)
  while go {
    logger.newline(1).loading("Waiting for events...");
    (3)
    let events = watcher.read_events()?;
    logger.info(format!("Got {} events", events.len()));
1 New up a logger and a stop variable
2 Loop until not go
3 Read events from the queue, otherwise block!

Handling detected events

    (1)
    for event in events {
      (2)
      let msg = format!("Event: {:?} for {:?}", event.mask, event.name);
      (3)
      logger.indent(1).log(msg);
    }
  }

  (4)
  watcher.rm_watch(watch)?;

  (5)
  Ok(go)
}
1 Loop over events
2 Make a nice message
3 Print it out
4 Clean up our watch just in case
5 All done!

Problems with this inotify example

  1. The go variable will always be true.

  2. It is an overly-broad watch (IN_ALL_EVENTS)!

  3. It doesn’t traverse the directory tree.

Try to ignore these. Work with me, here.

Ok. So.

Systems Programming!

It doesn’t have to be painful!

Recap: systems programming with Rust

  • Doesn’t have to feel low-level to be low-level.

  • Excellent ecosystem of crates.

  • Versatile interfaces to Linux functionality.

Packaging Rust binaries for Linux

Stuff we didn’t even cover

  • Command-line interfaces

    • The clap crate is exceptional

  • Notifications

    • Check out the notify_rust crate for great functionality

  • Async programming

  • Fault tolerance

    • The Bastion project looks really cool

  • Linux kernel integration with BPF/ePBF

  • Filesystem development

  • Containers

But Jonathan!

Have you ever written non-trivial things in Rust?

Yes. Lots.

Jonathan is the maintainer of several crates, including the testanything library for emitting test results in the Test Anything Protocol (TAP).

Enter: Hermione

Competent magic for your config files and more!

A package manager for your config files?

Hermione features

Current

  • Full Rust CLI

    • Portable across Linux, macOS, and Windows

  • Integrated package scaffolding and utilities

  • Package lifecycle hooks

Coming soon

  • Repositories

  • Self-contained package archive support

Soon ripping out git support in favor of package repositories and archive files.

Check us out at https://hermione.dev

Highly experimental!

I want to introduce co-maintainer Egli Hila

  • One of the best software engineers I know

  • Co-maintainer of Hermione

  • A real swell fella

  • Fantastic baker

Demo!

What you just saw

  • Command-line usage of Hermione

  • Hermione was used to install a package of config files

  • Config files were symlinked into place

Learning more about Rust

Learning more abot Hermione

Getting involved

If you are a Rustacean, we need help with cargo-appimage!

Thanks. End.