enge@inria

Tiny Build Farm for Guix, part 2

2025-10-24T00:00:00Z

Building science packages

In our efforts to create a Tiny Build Farm for Guix, that is supposed to report on the status of the packages assigned to the science team, so far we have seen how to set up the required infrastructure. On a dedicated machine with Guix as its operating system, we have added several Shepherd services: the Guix Build Coordinator together with a build agent; and the web server part of the BFFE, which enables us to follow the activity of the builders. For performance reasons, we have renounced at installing an instance of the Guix Data Service, and opt instead for talking to the instance operated by the Guix project at https://data.guix.gnu.org/, which continually evaluates the Guix master branch and creates derivations for all packages in the distribution. The next step is to explore how to programmatically talk to the remote data server from a Guile script, how to extract derivations we are interested in, and how to submit them for building to our instance of the build coordinator.

Getting information from the data service

We need to install the two packages guix-data-service and (for later use) guix-build-coordinator on the TBFG machine, which contain Guile libraries with the necessary functionality.

⚠ If installed into a user profile, both packages pull in the guix package as a propagated input, which prevents the user from updating it through guix pull. It is thus recommended to run

guix shell guile-next guix-data-service guix-build-coordinator

instead. At the time of writing, the guile package in Guix is at version 3.0.9, while the data service library requires guile-next, which is at version 3.0.10.

Let us open a Guile REPL and execute the following code (to ease copy-pasting, I omit the prompt of the REPL; lines starting with a $ sign and a number correspond to results).

$ guile
(use-modules (guix-data-service client))
(define my-data-service "https://data.guix.gnu.org/")

(define json
  (guix-data-service-request my-data-service
                             "repository/1/branch/master.json"))
json
$1 = (("revisions" . #((("data_available" . #f) ("commit-hash" . "cb47639a8081e8e2d651ad1612bbd1e482766469") …

The call to guix-data-service-request is equivalent to opening the URL https://data.guix.gnu.org/repository/1/branch/master.json, which executes the same query as the URL https://data.guix.gnu.org/repository/1/branch/master without the .json at the end, but it returns the result in JSON format. Moreover, the function call transforms the JSON into a Guile data structure through the guile-json library; in particular, JSON arrays become Guile vectors and JSON objects become Guile association lists, or alists for short (these are lists of key-value pairs, so brace yourself for lots of parentheses in a row). Thus parsing the result and extracting the information we are interested in amounts to unwrapping these successive layers; in true Scheme/Lisp style we will also usually transform the vectors into lists using the vector->list function. The JSON we asked for is an object with a unique field revisions, which contains an array of revisions, that is, git commits on the master branch; every revision is an object with the three fields date, commit-hash (these are strings) and data_available, a boolean indicating whether the data service has computed the derivations for this commit or not (which corresponds to the green or grey badges on the website). This structure can be derived by looking at and playing with the variables in the REPL, or probably more conveniently by opening the corresponding URL in a web browser, which should show the JSON in a special mode. We can now write a small function (or maybe two even smaller functions) that query the data service and return a list of revisions for which the data service has computed the derivations:

(define (data-available? revision)
  ;; Given a REVISION, check whether it has been treated by the
  ;; data service.
  (assoc-ref revision "data_available"))

(define (get-revisions data-service)
  ;; Query DATA-SERVICE for the list of revisions it has successfully
  ;; treated in the master branch.
  (filter data-available?
    (vector->list
      (assoc-ref
        (guix-data-service-request data-service
          "repository/1/branch/master.json")
        "revisions"))))

(define revisions (get-revisions my-data-service))
revisions
$2 = ((("data_available" . #t) ("commit-hash" . " …

In the following, we will work with revisions in this form, although mainly the commit hashes are of interest. We could print them as follows:

(define commits
  (map (lambda (revision)
         (assoc-ref revision "commit-hash"))
       revisions))
commits
$3 = ("b966f4007c8492ad89eedf32dd91b3352dba594e" "8a1f56cf8710fc142a2f8ef2e52be82e8aa9f53e" …
(length commits)
$4 = 46
(define commit (car commits))
commit
$5 = b966f4007c8492ad89eedf32dd91b3352dba594e

By default the data service returns 100 revisions (including those for which no data is available), which will be amply enough for our purposes.

The next step is to obtain the derivations for a given revision, say the newest one with data available. Again this is most easily reverse-engineered from the web interface of the data service: Click on the latest revision with a green badge, then on View package derivations; this shows how the URL is to be formed. Since we need all derivations, we also have to tick the All results checkbox; on the other hand, we may limit to one architecture, say x86_64-linux as System, and not consider cross-compilation by choosing (no target) for Target. These choices add GET parameters to the query, which can be passed as an alist for the optional third parameter of guix-data-service-request. Again adding .json to the URL (in front of the ?) shows the structure of the resulting JSON. It is then easy to end up with the following function; notice the use of the quasiquote ` and the unquote ,:

(define (get-derivations data-service commit system)
  ;; Query DATA-SERVICE for the list of derivations for the given COMMIT
  ;; and SYSTEM.
  (map
    (lambda (p)
      (assoc-ref p "derivation"))
    (vector->list
      (assoc-ref
        (guix-data-service-request data-service
          (string-append "revision/" commit "/package-derivations.json")
          `((system . ,system) (target . "none") (all_results . "on")))
        "derivations"))))

(define derivations
  (get-derivations my-data-service commit "x86_64-linux"))
(length derivations)
$6 = 29531
(car derivations)
$7 = "/gnu/store/000lxmn2d17bv2v6znvf6z5vi7ndy8q4-r-janeaustenr-1.0.0.drv"

So the derivations are simply strings pointing to files in the store (of the data service, so far they are not yet available on the TBFG machine).

Filtering out team packages

29000 derivations are more than our poor tiny machine can handle; the next step is to filter out those that correspond to packages in the science team. The team is responsible for certain package modules (or equivalently, for .scm files in the gnu/packages/ directory); which ones can be seen in the file CODEOWNERS checked into the Guix git repository, itself derived from etc/teams.scm. As it does not change very often, for simplicity we may determine the list of modules by hand, which may require us to resolve regular expressions (here: fortran(-.+|)) into lists of actually present modules; here we end up with the following:

(define my-locations
  '("algebra" "astronomy" "chemistry" "fortran-check" "fortran-xyz"
  "geo" "graph" "lean" "maths" "medical" "sagemath" "statistics"))

When starting the project, I had hoped to extract the interesting packages directly from the (strings representing) derivations, given a fixed list of package names. But it is a truth universally acknowledged that a programmer never has the singularly good fortune of such simplicity, whatever their feelings or views when first entering the neighbourhood of a problem. Here two reasons speak against it: First of all, the packages of a team may change over time as packages are added, removed or moved to a different module. More immediately, though, only the combination of package name and version can be easily recovered from the derivation by removing a fixed prefix, the hash and a fixed suffix, using the following function:

(define (derivation->name+version derivation)
  ;; Given a DERIVATION (by a string of the form "/gnu/store/..."),
  ;; return the part of it that encodes the name and the version
  ;; of the underlying package.
  (string-drop (basename derivation ".drv") 33))

Thus /gnu/store/000lxmn2d17bv2v6znvf6z5vi7ndy8q4-r-janeaustenr-1.0.0.drv becomes r-janeaustenr-1.0.0, which is the concatenation of the package name (which is mostly fixed over different revisions) and the package version (which usually increases over time) with a hyphen in-between. More often than not it is possible to guess the two components: Here they are r-janeaustenr and 1.0.0. Package names often contain hyphens (like here, they serve to separate a language part, r, and the upstream name, janeaustenr, see the Guix naming conventions); this could be handled by splitting at the last hyphen, but versions may also contain hyphens. Both can contain alphabetic and numeric components. Thus it would be quite possible that the above derivation is for the flourishingly named version janeaustenr-1.0.0 of the r package.

So we need more code to extract the desired information. Luckily the data service knows about the packages in a revision, with their names and their versions in different fields; and also about their locations, that is, the files in which they are defined.

(define (get-packages data-service commit)
  ;; Query DATA-SERVICE for the list of packages for the given COMMIT.
  (vector->list
    (assoc-ref
      (guix-data-service-request data-service
        (string-append "revision/" commit "/packages.json")
        `((field . "version") (field . "location") (all_results . "on")))
      "packages")))

(define packages (get-packages my-data-service commit))
(car packages)
$8 = (("location" ("column" . 2) ("line" . 8273) ("file" . "gnu/packages/games.scm")) ("version" . "0.27.1") ("name" . "0ad"))

It is now enough to compare the file name with our list of locations to extract the packages we are interested in.

(define (location-package? package locations)
  ;; Check whether the PACKAGE comes from the list of LOCATIONS.
  (let* ((file (assoc-ref (assoc-ref package "location") "file"))
         (module (basename file ".scm")))
        (member module locations)))

(use-modules (srfi srfi-26))
(define (packages-name-version data-service commit locations)
  ;; Query DATA-SERVICE for a list of packages for the given COMMIT
  ;; that come from the list of LOCATIONS. Return a list of two-element
  ;; lists with the names and versions of these packages.
  (map
    (lambda (package)
      (list (assoc-ref package "name") (assoc-ref package "version")))
    (filter
      (cut location-package? <> locations)
      (get-packages data-service commit))))

(define team-name-versions
  (packages-name-version my-data-service commit my-locations))
(car team-name-versions)
$9 = ("4ti2" "1.6.12")

Finally we just need to compare the extracted team package names and their versions with the derivations. Unfortunately this can be quite costly; the following code presents a somewhat optimised solution with memory usage linear in the result, but a quadratic number of comparisons (thanks to Liliana Prikler for suggesting the use of filter-map to me):

(use-modules (srfi srfi-1))
(define (special-cartesian-product X Y)
  ;; Let X and Y be lists of two element lists of the form (x z) and (y z),
  ;; respectively. Return a list of all the (x y) such that there is an
  ;; element z with (x z) in X and (y z) in Y.
  (fold cons '()
        (filter-map (lambda (xz)
                      (let ((yz (find (lambda (yz)
                                        (equal? (cadr xz) (cadr yz)))
                                      Y)))
                        (if yz
                            (list (car xz) (car yz))
                            #f)))
                    X)))

(define (team-derivations data-service commit system locations)
  ;; Query DATA-SERVICE for the list of derivations for the given COMMIT
  ;; and SYSTEM, filtered by the LOCATIONS of the packages.
  ;; To memorise the computed information, return a list of two element
  ;; lists, each containing a derivation and the corresponding name.
  (let* ((derivations (get-derivations data-service commit system))
         (X (map
              (lambda (d)
                (list d (derivation->name+version d)))
            derivations))
         (name-versions
           (packages-name-version data-service commit locations))
         (Y (map
              (lambda (nv)
                (list (car nv) (string-append (car nv) "-" (cadr nv))))
            name-versions)))
    (special-cartesian-product X Y)))

(define (sort-derivation-names derivation-names)
  ;; Just for the fun of it, sort DERIVATION-NAMES, a list of two element
  ;; lists containing derivations and their names, by names.
  (sort derivation-names
        (lambda (x y)
          (string<? (cadr x) (cadr y)))))

(define good-derivation-names
  (sort-derivation-names
    (team-derivations my-data-service commit "x86_64-linux" my-locations)))
(define derivation-name
        (find (lambda (dn)
                (equal? (cadr dn) "lrslib"))
              good-derivation-names))
derivation-name
$10 = ("/gnu/store/3pxq1g2java4f8nwfq7n98qjvhkr1b34-lrslib-7.2.drv" "lrslib")

Strictly speaking, the function team-derivations is not correct; if there were simultaneously a derivation for the package r-jauneaustenr at version 1.0.0 and a derivation for the package r at version jauneaustenr-1.0.0, then either both or none of them would match, while it is possible that only one of the packages is covered by the science team, a situation not yet encountered; at worst, we would capture one too many derivations. For testing purposes during the development of the TBFG, we additionally check whether the name equals lrslib; in this way only one derivation is returned (while at the time of writing there are more than 700 packages covered by the science team). Moreover the package in question is a self-contained C program (without any inputs), which compiles rather quickly.

Submitting builds

Now that we have a list of derivations, we would like to submit them from our Guile script to the build coordinator. This is not very different from the approach seen last time for submitting from the command line. Again it is recommended to open a browser window on the /activity page of the BFFE to see the build coordinator and the agent in action.

(use-modules (guix-build-coordinator client-communication))

(define my-build-coordinator "http://localhost:8746")
(define ignore-if-build-for-derivation-exists? #f)
(define ignore-if-build-for-outputs-exists? #f)
(define ensure-all-related-derivation-outputs-have-builds? #f)
(define priority 0)

(define (submit-build build-coordinator data-service derivation tags)
  ;; Given a DERIVATION (as a string), submit it to BUILD-COORDINATOR
  ;; together with TAGS;
  ;; DATA-SERVICE is passed through and used by the build coordinator to
  ;; obtain the derivation file and further references contained in
  ;; DERIVATION.
  (send-submit-build-request
    build-coordinator derivation (list data-service) 0 priority
    ignore-if-build-for-derivation-exists?
    ignore-if-build-for-outputs-exists?
    ensure-all-related-derivation-outputs-have-builds?
    tags))

(submit-build my-build-coordinator my-data-service (car derivation-name) '())
$11 = (("build-submitted" . "8f8f1cad-fe9c-462c-bc59-3d1f87abf942"))
$12 = #<<response> …

The global variables, which we pass on to the submit-build function, determine the behaviour of the build coordinator. If ignore-if-build-for-derivation-exists? is true, then the build will not be carried out a second time if it was already tried (successfully or not) by the build coordinator before. In production, it will thus be preferable to set it to #t; while still experimenting, we are likely to submit the same derivation several times. Setting the value to #f would also make sense to check that rebuilding the same package works. The variable ignore-if-build-for-outputs-exists? goes a bit further; if set to #t, then the build will not be carried out if a different derivation with the same output was already tried (a very technical distinction; I would recommend to leave it at #f). If ensure-all-related-derivation-outputs-have-builds? is #t, then the build coordinator will recursively submit builds for all the derivations required as inputs to a given derivation. While this sounds reasonable at first, it can go very far, since the coordinator does not look at the store, but at the builds it has handled itself and recorded in its database. This means that the first build submission, when the database is still empty, will entail a complete bootstrap of the Guix distribution. So I would recommend to leave it also at #f. Then the build works as follows: The coordinator sends the derivation to an agent. The agent tries to download all required inputs from a substitute server and if successful, will build only the derivation it is asked to build. Otherwise, it reports back to the coordinator that it has encountered a set-up failure, together with a list of missing inputs. This triggers a hook in the coordinator, and the default hook is to add the missing inputs to the list of outstanding builds, as well as the failed build itself to try it again once the inputs are available. In this way, even if ensure-all-related-derivation-outputs-have-builds? is #f, all really missing inputs will be built recursively, until the build succeeds or a real failure in one of its inputs is encountered.

The submission immediately returns two values, without waiting for the package build to finish. The first return value can be used to link the submitted derivation to the shown UUID of the build, which is a key in the build coordinator database. The second return value is the HTTP response, which we will ignore from now on.

Tags can be added in a parenthesis rich format; the parameter is a list of tags, where each tag is a two element list (not a pair!), in which both elements are pairs. The first one pairs the keyword key to a value, the second one pairs the keyword value to a value (the values are used to construct the URL and can be strings or numbers). So the following would work:

(define tags `(((key . "commit")(value . ,commit))
               ((key . "name")(value . ,(cadr derivation-name)))
               ((key . "build")(value . 2))))
(submit-build my-build-coordinator my-data-service (car derivation-name) tags)
$13 (("build-submitted" . "82a56cac-1e93-4b4a-926f-d8762f919219"))
$14 = #<<response> …

The tags are shown in the activity window and are also recorded in the build coordinator database; as shown here, they can encode arbitrary additional information of a build, such as the commit it comes from, the package name or the submission count for a given derivation.

Code

For ease of use, the code developed in this post is made available, under GPLv3 or later, in a dedicated git repository on Codeberg. More precisely, it is collected in the file tbfg.scm at commit 51eb5c6d45c66d15b7c14340ec3af0732b5b66fd.

Outlook

We have queried the data service and used the resulting information on packages and derivations to submit build jobs to the build coordinator. But so far we have no programmatical access to the build results; we only saw the builds flicker by on the BFFE website. It would be nice to record success or failure, and more generally to keep track of the builds; this will be our next step. Since we do not want to operate a substitute server, but rather follow the state of the packages under the responsibility of the science team, unlike the official build farms we are not necessarily interested in obtaining the build results. These are sent from the build agents to the build coordinator; on the bordeaux build farm the nar herder shovels them to a separate substitute server. For us everything is on the same machine, which will thus contain successfully built packages in its store (at least until the next guix gc run). If desired, these could be made available using guix publish.

Tiny Build Farm for Guix, part 1

2025-08-27T00:00:00Z

Setting up scores of services

One of the oft-cited reasons people give for not switching to Guix is that their favourite software is too outdated, and a look at Repology shows that they are not wrong. Now the number of active committers in the Guix project is amazingly small, and even counting all contributors I am impressed by what these few people actually achieve. Nevertheless I wondered how I could improve the situation at least a little bit for packages I am interested in, that is, for the science team; and the first step is to get an account of what actually builds and what does not. So I decided to set up my own little build farm, limited to the packages in the scope of the science team, using the same technology that powers the bordeaux build farm. I call it the Tiny Build Farm for Guix, or TBFG for short, and this post is the first one in (hopefully) a series of blog posts about the topic; at the time of starting this series, the TBFG does not actually exist yet, so wish me luck.

Motivation

Before trying to solve a technical problem, let me digress a little bit: Is there actually a problem? And if yes, why? As has become my conviction over the years, the really difficult and major problems in a project such as Guix are actually social and not technical. They are rooted in the structure of Guix as a loosely coupled group of volunteers who work on a common goal, mostly in their spare time; but when I speak about a common goal, every volunteer has in fact their own goals, and arriving at a coherent whole is partially due to the internal structuring of the social project, and partially an emerging property of a complex system. Concretely, it happens often that contributors propose a package for a software project they like; if it concerns free software and follows our packaging guidelines, it usually ends up being committed to the software distribution. The original contributor may leave, the package may bitrot and stop being buildable due to changes made in other parts of the distribution. Sometimes people submit a bug report, sometimes committers without interest in the actual software provide a fix, but maybe nobody uses the software anymore, and it happens that over several years nobody notices it is broken. This is problematic since even broken packages use resources on the build farms. And when introducing, say, an update to a library package, it becomes difficult to say whether the failure of a dependency is a new phenomenon due to the change or whether it was already present. So there are good reasons to strive for a distribution that is 100% buildable at all times. And while I alone certainly cannot reach this for the currently more than 28000 packages in Guix, doing it only for the science team, or maybe only the algebra and maths modules, in which I am particularly interested, appears to be a reachable goal.

But is a tiny build farm really needed? The honest answer is “no”, since the information is already out there in the big build farms. For historical reasons, Guix has two of them. One is CI, also called berlin for the location of most of its build machines; it relies on Cuirass, a continuous integration system written purposefully for Guix. It shows the state of the master branch and provides a dashboard from which the desired information could certainly be extracted automatically. On the other hand there is the bordeaux build farm, named after the location of its head node; it runs a suite of continuous integration tools written purposefully for Guix by Christopher Baines. One of its parts is the Guix Data Service, and its REST API with a JSON frontend makes it again possible to extract the information I am interested in – a system that knows about all packages in Guix by definition knows about the packages in the realm of the science team.

So setting up my TBFG is mainly an educational project – I would like to learn how our technology works. But the TBFG can also be used to obtain information about a collection of packages that is not part of Guix proper, or to look at the impact of local changes. Many people and projects have successfully set up Cuirass to manage their local package collection; this is, for instance, the case for the Guix Science and Guix HPC projects through the build server at INRIA Bordeaux. The software behind the bordeaux build farm is more complex and consists of several interconnected Shepherd services, so that it may in fact be less suited for a personal project. However, not least because I host part of that build farm at home, I am more interested in understanding this software stack, which is also less documented. So I am going to build the TBFG on top of this technology. Before diving in, I take the opportunity to thank Christopher Baines for his precious help during a Guix/Nix hackers' meeting; without him, I would not have been able to launch myself into this endeavour.

Build coordinator and agent

As a first step, we need to install the Guix Build Coordinator and one or more build agents. For a really tiny TBFG, I will keep everything on only one machine set aside for the purpose; it is called bedok and is one of the Lenovo Thinkpad X1 Gen9 with a four core 11th Gen Intel Core i7-1165G7 processor running at 2.80GHz graciously donated by Tweag. For the build coordinator, this is trivial; simply add the two lines

(service guix-build-coordinator-service-type
  (guix-build-coordinator-configuration))

to the services configuration of the Guix system declaration and reconfigure the machine. For a start, the default configuration options explained in more detail in the manual are appropriate (things will become more complicated if anything is to be done with the build results, which would require to set the hooks field of the configuration record). See also the documentation in the git repository of the project.

Next we need the guix-build-coordinator package, which provides a command line interface to the coordinator. It could be installed into an arbitrary profile since the security model of the build coordinator is very basic: It is assumed that the coordinator runs on a server of its own, and anybody with access to the machine has control over the service.

⚠ However installing the package into a user profile currently has a big drawback: It propagates the guix package, and the corresponding guix takes precedence over the one in $HOME/.config/guix/current/bin. The latter one is updated by guix pull, but the former one is not; so without a special approach, this prevents the user from updating Guix. Instead one can start a Guix shell as follows:

guix shell guix-build-coordinator

Running

$ guix-build-coordinator agent list

shows nothing, so the next step is to set up a build agent, which requires some preparations on the machine where the build server is running. Executing

$ guix-build-coordinator agent new
e092df28-3418-4f94-b7f0-a214b03291ee

creates and prints a new random version 4 UUID and stores it in the agents table in its internal database. The next step is to set up authentication; for a small number of build agents, a password file is a suitable approach, otherwise an authentication token, which may be shared among several agents, can be used. So we create a password for the agent by running

$ guix-build-coordinator agent e092df28-3418-4f94-b7f0-a214b03291ee password new
new password: IUwGrsEklfu0kVf_QquCOdzfa6-P52qPlcwBd5YB

which again prints the password and saves it into the coordinator database.

To simplify things for the human brain, we can give the agent a name; since there is not yet a command line argument for this, we take it as a pretense to look more closely at the SQLite database structure. So install the sqlite package into the root profile, launch sqlite3 on the coordinator database, and run the following commands:

# sqlite3 /var/lib/guix-build-coordinator/guix_build_coordinator.db
sqlite> .tables
agent_passwords
agent_tags
agents
…
builds
…
tags
…
sqlite> select * from agent_passwords;
1|e092df28-3418-4f94-b7f0-a214b03291ee|IUwGrsEklfu0kVf_QquCOdzfa6-P52qPlcwBd5YB|2025-08-13 00:00:00
sqlite> .schema agents
CREATE TABLE agents (
       id TEXT PRIMARY KEY,
       description TEXT
, name TEXT, active NOT NULL DEFAULT 1);

There are quite a few tables, but for now we are mainly interested in those related to the agents. We can recover the password in case we forgot to write it down (alternatively we could run guix-build-coordinator agent e092df28-3418-4f94-b7f0-a214b03291ee password), and we see that agents can have a name and a description, and be active or not. So still in SQLite run

sqlite> update agents set name='bedok', description='TBFG agent' where id='e092df28-3418-4f94-b7f0-a214b03291ee';
sqlite> select * from agents;
e092df28-3418-4f94-b7f0-a214b03291ee|TBFG agent|bedok|1

Alternatively, to check that everything has gone well, run (not necessarily as root anymore):

$ guix-build-coordinator agent list
e092df28-3418-4f94-b7f0-a214b03291ee: bedok
  description:
  TBFG agent  active?: true
  0 allocated builds:
  requested systems:
  tags:

Now it is finally time to really set up the build agent! On the machine where it is supposed to run (in our case, this is bedok again, but it could be an arbitrary machine somewhere on the Internet, since all communication would take place over https), we need to handle the password. As usual in Guix, secrets are not saved in configuration that is publicly visible in the store, but rather as separate state; so as root create a file /etc/guix-build-coordinator/agent-bedok-passwd containing the password IUwGrsEklfu0kVf_QquCOdzfa6-P52qPlcwBd5YB created above by the coordinator. Then add the following snippet to the server part of the operating system configuration:

(service guix-build-coordinator-agent-service-type
  (guix-build-coordinator-agent-configuration
    (authentication
      (guix-build-coordinator-agent-password-file-auth
        (uuid "e092df28-3418-4f94-b7f0-a214b03291ee")
        (password-file
          "/etc/guix-build-coordinator/agent-bedok-passwd")))
    (derivation-substitute-urls
      '("https://data.guix.gnu.org"))
    (non-derivation-substitute-urls
      '("https://bordeaux.guix.gnu.org"))
    (systems '("x86_64-linux" "i686-linux"))
    (max-parallel-builds 4)
    (max-parallel-uploads 2)
    (max-1min-load-average 6)))

and reconfigure the machine. Concerning the different parameters, see the documentation. For authentication, we need to provide our uuid and the location of the password-file. Since the agent runs on the same machine as the coordinator, I kept the coordinator field at its default "http://localhost:8745"; otherwise localhost needs to be replaced by the host name of the coordinator, and the protocol should be set to https. The derivation-substitute-urls field has no default; we will discuss it in the next section. The non-derivation-substitute-urls field also needs to be set to avoid compiling each and every package input locally; here I chose to only use the bordeaux build farm, but one could add "https://ci.guix.gnu.org" to also fetch packages from berlin. If the system field is not set, then only packages for the system on which the agent is running (most likely x86_64-linux) are handled; here it is useful to add i686-linux. Or on an ARM machine, the combo '("aarch64-linux" "armhf-linux") makes sense. The numerical parameters can be left at their defaults; here I am trying to limit the load on my four core processor.

If all goes well, we should see lines in the agent logfile /var/log/guix-build-coordinator/agent.log looking like

2025-08-13 00:00:00 (INFO ): starting agent e092df28-3418-4f94-b7f0-a214b03291ee
2025-08-13 00:00:00 (INFO ): connecting to coordinator http://localhost:8745
2025-08-13 00:00:00 (INFO ): running 0 threads, currently allocated 0 builds
2025-08-13 00:00:00 (INFO ): starting 0 new builds

and running

$ guix-build-coordinator agent list

on the coordinator machine again should now print two entries beneath requested systems.

As a last step, get back to the /etc/guix-build-coordinator/agent-bedok-passwd file, which is probably world-readable. Starting the build agent has created a user guix-build-coordinator-agent, and I would recommend to have the file be owned by that user, and remove permissions from all other users.

Data service

This is the elephant in the room, almost literally. When starting my project of the TBFG, I had intended to set up the full stack of software needed to run the bordeaux build farm, and the Guix Data Service is a very important part of it. But the code base is massive (more than 30000 lines of Scheme code at the time of writing), and also the required ressources are massive: The service spends its time polling the git repository of Guix, compiling the sources and computing all the derivations for all the packages, which are then stored in a database. Rinse and repeat for the next commit. (In reality, the data service does even more, in particular it also queries build servers and stores information about builds; but the above functionality is everything we need for the TBFG.) As can be seen, not even the official data service, running continuously on a powerful server, manages to do that for all commits: Some of them are marked as green, others, the grey ones, are skipped, in particular at times of high commit activity.

So I have decided to rely on the central Guix data service by configuring the corresponding derivation-substitute-urls field of the build coordinator as '(https://data.guix.gnu.org). All information obtained by clicking through the data service website can also be obtained as JSON through its REST API, which we will use in our scripts to determine the derivations of science team packages to be built.

For this to work, we also need the Guix daemon to accept the signing key of the data service; so the services field of the operating system declaration should look like this:

(services
  (append
    (modify-services %base-services
      (guix-service-type config =>
        (guix-configuration
          (substitute-urls '("https://bordeaux.guix.gnu.org"))
          (authorized-keys
            (list
              (local-file "keys/guix/bordeaux.guix.gnu.org-export.pub")
              (local-file "keys/guix/data.guix.gnu.org.pub")))
          (max-silent-time (* 24 3600))
          (timeout (* 48 3600)))))
    (list
      (service guix-build-coordinator-service-type
        (guix-build-coordinator-configuration))
      …)))

where the key files are copy-pasted into the local keys/guix subdirectory from the corresponding place in the guix/maintenance git repository.

BFFE – Build Farm Front End

We could start submitting build jobs now, but to visualise what is happening, we need another service, bffe. The build farm frontend actually serves two purposes: On one hand on the bordeaux build farm, it submits build jobs for the master branch to ensure continuous substitute availability (and a different service, qa-frontpage, submits build jobs for testing branches and pull requests to the same build coordinator instance). On the other hand, it provides a web server that connects to the build coordinator and shows information about its status. We will only need the second functionality. For this, add the following snippet to the service configuration of the TBFG machine:

(service bffe-service-type
  (bffe-configuration
    (arguments
      #~(list
        #:web-server-args
          '(#:event-source "http://localhost:8746"
            #:controller-args (#:title "Science team build farm"))))))

We use the #:web-server-args argument and provide as event-source the local build coordinator instance, which communicates with clients on port 8746 (while communication with the agents runs on port 8745 as seen above). For more details on the optional #:build argument, see the documentation in the Guix manual or the source code.

Reconfigure the system and open the BFFE website, either at http://localhost:8767 locally on the TBFG machine, or at http://192.168.1.80:8767 from another machine in your local network, where you have to adapt the IP address to your situation. The result is a rather empty web page, but it should at least show the title we have chosen. For our purposes, we are interested in the page obtained by appending /activity to the URL. This shows a box Recent activity, which is rightfully empty; and a list of agents per architecture and their current occupation, which should also be void, except possibly for the percentage giving the CPU load in case the agent also has other business. Clicking on the name of an agent sends us to yet another page with more details (among which the description we provided previously) .

Submitting a build

After all these preparations, we can finally submit our first build job! For this, keep the /activity page open, and run

$ DRV=`guix build hello --derivations`
$ guix-build-coordinator build --derivation-substitute-urls=https://data.guix.gnu.org $DRV
build submitted as 7bdd2249-1214-431a-aa61-de4d907f1b32

Providing a URL from which to fetch derivations is necessary because the build coordinator receives only the store path of the derivation and not the actual file; the same then recursively holds for inputs referenced in the submitted derivation. This could be a global parameter of the guix-build-coordinator-configuration, but currently needs to be specified by hand and can thus vary over time or depending on the build.

If all goes well, a UUID for the build is printed in the terminal, and three lines appear on the BFFE web page in the Recent activity box: Build submitted, Build started and Build succeeded. The /var/log/guix-build-coordinator/coordinator.log and /var/log/guix-build-coordinator/agent.log files should also contain matching information. And you can go to the http://192.168.1.80:8767/build/7bdd2249-1214-431a-aa61-de4d907f1b32 URL to see more detailed information on the build, with potentially a link to the build log.

This link unfortunately does not work out of the box, and some more configuration is required to make the build logs available.

Nginx for the build logs

One possibility for accessing the build logs is by directly looking them up in the place where they are stored by the build coordinator, the directory /var/lib/guix-build-coordinator/build-logs/. Each build corresponds to a subdirectory named after its UUID and containing the file log.gz.

Alternatively, we can imitate the behaviour of the bordeaux build farm and set up a separate nginx web server using the following service snippet:

(service nginx-service-type
  (nginx-configuration
    (server-blocks
      (list
	(nginx-server-configuration
	  (listen '("80" "[::]:80"))
	  (locations
	    (list
	      (nginx-location-configuration
		(uri "~ \"\\/build\\/([a-z0-9-]{36})/log$\"")
                (body '("alias /var/lib/guix-build-coordinator/build-logs/$1/log;"
                        "add_header Content-Type 'text/plain; charset=UTF-8';"
                        "gzip_static always;"
                        "gunzip on;")))
              (nginx-location-configuration
                (uri "/")
                (body '("proxy_pass http://localhost:8767;"
                        "proxy_http_version 1.1;"
                        "proxy_set_header Connection \"\";")))
              (nginx-location-configuration
                (uri "/events")
                (body '("proxy_pass http://localhost:8767;"
                        "proxy_http_version 1.1;"
                        "proxy_buffering off;"
                        "proxy_set_header Connection \"\";"))))))))))

The first nginx-location-configuration serves the build logs, while the other two are reverse proxies towards the pages provided by BFFE at port 8767. If the activity page is now accessed through the nginx web server at the standard port through the URL http://192.168.1.80/activity, it presents links to builds and their log files, which can be clicked on, and the uncompressed log files are shown directly in the browser.

Outlook

This was a lot of work for setting up the necessary services! But hopefully it was also a good occasion to understand how the different components interact. In the next installment we will start writing our own scripts to communicate with these services; in particular we will work with the data service to retrieve information about the packages we are interested in.

Wireguard VPN with Guix

2025-08-07T00:00:00Z

Needing a VPN

Recently I changed my ISP, and the new one uses Carrier-grade NAT, or CGNAT, by default. While this sounds fancy and professional, it is in fact even worse than conventional NAT: Not only do all my devices share the same IPv4, but I share one IPv4 with several other customers! Apparently I am only assigned a few out of the 65535 ports, and this assignment may change from day to day, which implies that I cannot connect from the outside to any of my home devices. However, I do have a separate IPv4 of my own for a virtual machine at Aquilenet, and it should be possible to use this as a trampoline to access my home through a virtual private network. We are already employing WireGuard for one of the Guix build farms, so it felt like a natural choice. Guix provides the wireguard-service-type, which is documented with all its options in the manual; but without an explanation of the general concepts behind the service it is a bit difficult to set up. The Guix Cookbook has an entry on WireGuard, but it is concerned with kernel modules and connecting to an existing WireGuard VPN, while my goal was to set one up in the first place. This turned out to be surprisingly easy.

The wireguard-tools package comes with an executable wg, and running wg --help is enough to guess how WireGuard works; essentially we need the following two subcommands:

genkey: Generates a new private key and writes it to stdout
pubkey: Reads a private key from stdin and writes a public key to stdout

Unlike other VPN, WireGuard appears to be more symmetric in the sense that it does not distinguish between servers and clients; to talk to each other, two participants just need to create a pair of public and private keys each, and then to be made aware of the other's public key. In our asymmetric situation in which only one of them has a public IPv4, we will nevertheless distinguish the server, which is reachable from everywhere thanks to its IP, and the clients hidden behind the CGNAT.

Creating key pairs

In a first step, we create a private key for the server. In Guix, secrets are (so far) not handled through the world-readable store, but as state directly on the machine, and wireguard-service-type expects by default the private key in the file /etc/wireguard/private.key. So we connect as root to the server machine and execute

mkdir /etc/wireguard
umask 077
wg genkey > /etc/wireguard/private.key

The call to umask is needed (at least with my shell settings) to placate the WireGuard warning that the private key file is world-readable, which indeed defies its purpose. The file contains a short base64 encoded number such as GEhlpFGslXfo9We9jhrXham4LztmqSmpdE4ivML4qXc=. Given the size (or rather lack thereof) of this number, it looks like WireGuard uses elliptic curve cryptography with a fixed elliptic curve and a fixed basepoint of 256 bits, so with a security level of 128 bits.

In a second step, we need to create the corresponding public key using the command

wg pubkey < /etc/wireguard/private.key

This outputs a base64 encoded number of similar size; in our example, AFL8UecS3GFX3hK8e6yWOK4s5RVrTpvTq2A0pdGuylQ=. This public key is in fact not needed by the server, but only by the clients wishing to connect to it (following a basic principle of asymmetric cryptography), so we need to stow it away in a file. But since the process of deriving the public key from a private key is deterministic, we may actually forget the public key and recreate it when needed. And notice that the public key is so short that it could even be exchanged on a postcard or over the phone.

This process of creating a key pair needs to be repeated on each client, or more generally, each participant in the VPN. Let us assume we have one client with private key 0G4uhLLeY5NYmg/FobRB0p75wMrGwmmzhuoAdfX243I= in /etc/wireguard/private.key and corresponding public key BgMzEZPUGAtbSqVPRdzgdLVhAPMLaOzHe7uNFAMVLCk=.

Setting up the server

Each participant in the WireGuard network uses a private IPv4, usually from the 10.0.0.0/8 range. We give 10.0.0.1 to the server and 10.0.0.2 to the client, and can now follow the documentation (you need to scroll down a bit) of wireguard-service-type to write the corresponding block in the Guix operating system configuration of the server:

(service wireguard-service-type
  (wireguard-configuration
    (addresses '("10.0.0.1/32"))
    (peers
      (list
        (wireguard-peer
          (name "client")
          (public-key "BgMzEZPUGAtbSqVPRdzgdLVhAPMLaOzHe7uNFAMVLCk=")
          (allowed-ips '("10.0.0.2/32")))))))

The addresses field is in fact the default; there is also an optional port field with default 51820. The peers field is a list of, well, peers in the VPN which are allowed to connect to the server (so it is in theory possible to create strange connection graphs); in this case, we register only one client peer with an arbitrary name, its public key created above, and its assigned private IP address. That is all! Now we can guix system reconfigure, and the server is ready.

Setting up the client

As stated above, in principle there does not seem to be a distinction between clients and server in WireGuard, so the operating system declaration on the client is similar to that on the server. But I think that nevertheless, it is necessary to bootstrap the network topology. And in our case, the inherent distinction between the server machine which is, say, publicly reachable on the IPv4 198.51.100.0 under the name vpn.example.org, and the client hidden by CGNAT needs to be taken into account. So we need the client to punch a hole into the NAT and to reach out to the server, which leads to the following service block:

(service wireguard-service-type
  (wireguard-configuration
    (addresses '("10.0.0.2/32"))
    (peers
      (list
        (wireguard-peer
          (name "server")
          (public-key "AFL8UecS3GFX3hK8e6yWOK4s5RVrTpvTq2A0pdGuylQ=")
          (allowed-ips '("10.0.0.1/32"))
          (endpoint "198.51.100.0:51820")
          (keep-alive 60))))))

The first fields are symmetric to the corresponding fields on the server. But the additional endpoint and keep-alive fields tell the client to connect to the server on its public IPv4 address (and the default port 51820) for the initial handshake establishing the session, and to keep it alive by reconnecting every 60 seconds. I have tried to use the host name vpn.example.org instead of the IPv4, but this ended up being resolved to an IPv6, which did not work. So guix system reconfigure the client, wait for at most one minute, and the VPN is running!

Looking behind the scenes

The following is not necessary for setting up the VPN, but it may be helpful for trouble shooting; and I was curious to see how the VPN manifested itself. Running ifconfig as root on the client, say, shows a new interface

wg0 Link encap:(hwtype unknown)
    inet addr:10.0.0.2  P-t-P:10.0.0.2  Mask:255.255.255.255
…

and running wg (again as root) shows the information about the VPN that we entered into the service description:

interface: wg0
  public key: BgMzEZPUGAtbSqVPRdzgdLVhAPMLaOzHe7uNFAMVLCk=
  private key: (hidden)
  listening port: 51820

peer: AFL8UecS3GFX3hK8e6yWOK4s5RVrTpvTq2A0pdGuylQ=
  endpoint: 198.51.100.0:51820
  allowed ips: 10.0.0.1/32
  latest handshake: 1 minute, 15 seconds ago
  transfer: 555.77 KiB received, 1.38 MiB sent
  persistent keepalive: every 1 minute

Finally, connecting from the outside!

But let us not get carried away by the beauty of technology (and cryptography), but get back to our initial concern: Connect from the outside to the machine in the home network, which we know under the name of client. This is now just a matter of two hops with ssh: First do an ssh vpn.example.org, and once on the VPN server machine, a second ssh 10.0.0.2. This can be automated by the following entry in .ssh/config on the machine from which we try to connect:

Host client
  Hostname 10.0.0.2
  ProxyJump vpn.example.org

so that from now on, ssh client will send us into the home network. Voilà, problem solved!

Epilogue

After installing my WireGuard VPN, I talked with a fellow geek from Aquilenet, who has the same ISP, and who suggested an alternative solution to me. I should call the hotline and pronounce the magic words that I would like an “IP rollback” so that servers at home become accessible. This helps passing the barrier of the first support level. The next level then initiates the “rollback”, which means going from IPv6 (plus IPv4 with CGNAT) back to regular IPv4 (without IPv6). After a few hours or days, the new more or less static (not guaranteed to be so, but actually not changing) IPv4 address is established, and IPv6 is disabled in the wireless router provided by the ISP. One can then reenable IPv6 in the router and lives in the best of all worlds – with a static IPv4 address, IPv6 and a WireGuard VPN on top of it all.