Setting up scores of services

One of the oft-cited reasons people give for not switching to Guix is that their favourite software is too outdated, and a look at Repology shows that they are not wrong. Now the number of active committers in the Guix project is amazingly small, and even counting all contributors I am impressed by what these few people actually achieve. Nevertheless I wondered how I could improve the situation at least a little bit for packages I am interested in, that is, for the science team; and the first step is to get an account of what actually builds and what does not. So I decided to set up my own little build farm, limited to the packages in the scope of the science team, using the same technology that powers the bordeaux build farm. I call it the Tiny Build Farm for Guix, or TBFG for short, and this post is the first one in (hopefully) a series of blog posts about the topic; at the time of starting this series, the TBFG does not actually exist yet, so wish me luck.

Motivation

Before trying to solve a technical problem, let me digress a little bit: Is there actually a problem? And if yes, why? As has become my conviction over the years, the really difficult and major problems in a project such as Guix are actually social and not technical. They are rooted in the structure of Guix as a loosely coupled group of volunteers who work on a common goal, mostly in their spare time; but when I speak about a common goal, every volunteer has in fact their own goals, and arriving at a coherent whole is partially due to the internal structuring of the social project, and partially an emerging property of a complex system. Concretely, it happens often that contributors propose a package for a software project they like; if it concerns free software and follows our packaging guidelines, it usually ends up being committed to the software distribution. The original contributor may leave, the package may bitrot and stop being buildable due to changes made in other parts of the distribution. Sometimes people submit a bug report, sometimes committers without interest in the actual software provide a fix, but maybe nobody uses the software anymore, and it happens that over several years nobody notices it is broken. This is problematic since even broken packages use resources on the build farms. And when introducing, say, an update to a library package, it becomes difficult to say whether the failure of a dependency is a new phenomenon due to the change or whether it was already present. So there are good reasons to strive for a distribution that is 100% buildable at all times. And while I alone certainly cannot reach this for the currently more than 28000 packages in Guix, doing it only for the science team, or maybe only the algebra and maths modules, in which I am particularly interested, appears to be a reachable goal.

But is a tiny build farm really needed? The honest answer is “no”, since the information is already out there in the big build farms. For historical reasons, Guix has two of them. One is CI, also called berlin for the location of most of its build machines; it relies on Cuirass, a continuous integration system written purposefully for Guix. It shows the state of the master branch and provides a dashboard from which the desired information could certainly be extracted automatically. On the other hand there is the bordeaux build farm, named after the location of its head node; it runs a suite of continuous integration tools written purposefully for Guix by Christopher Baines. One of its parts is the Guix Data Service, and its REST API with a JSON frontend makes it again possible to extract the information I am interested in – a system that knows about all packages in Guix by definition knows about the packages in the realm of the science team.

So setting up my TBFG is mainly an educational project – I would like to learn how our technology works. But the TBFG can also be used to obtain information about a collection of packages that is not part of Guix proper, or to look at the impact of local changes. Many people and projects have successfully set up Cuirass to manage their local package collection; this is, for instance, the case for the Guix Science and Guix HPC projects through the build server at INRIA Bordeaux. The software behind the bordeaux build farm is more complex and consists of several interconnected Shepherd services, so that it may in fact be less suited for a personal project. However, not least because I host part of that build farm at home, I am more interested in understanding this software stack, which is also less documented. So I am going to build the TBFG on top of this technology. Before diving in, I take the opportunity to thank Christopher Baines for his precious help during a Guix/Nix hackers' meeting; without him, I would not have been able to launch myself into this endeavour.

Build coordinator and agent

As a first step, we need to install the Guix Build Coordinator and one or more build agents. For a really tiny TBFG, I will keep everything on only one machine set aside for the purpose; it is called bedok and is one of the Lenovo Thinkpad X1 Gen9 with a four core 11th Gen Intel Core i7-1165G7 processor running at 2.80GHz graciously donated by Tweag. For the build coordinator, this is trivial; simply add the two lines

(service guix-build-coordinator-service-type
  (guix-build-coordinator-configuration))

to the services configuration of the Guix system declaration and reconfigure the machine. For a start, the default configuration options explained in more detail in the manual are appropriate (things will become more complicated if anything is to be done with the build results, which would require to set the hooks field of the configuration record). See also the documentation in the git repository of the project.

Next we need the guix-build-coordinator package, which provides a command line interface to the coordinator. It could be installed into an arbitrary profile since the security model of the build coordinator is very basic: It is assumed that the coordinator runs on a server of its own, and anybody with access to the machine has control over the service.

⚠ However installing the package into a user profile currently has a big drawback: It propagates the guix package, and the corresponding guix takes precedence over the one in $HOME/.config/guix/current/bin. The latter one is updated by guix pull, but the former one is not; so without a special approach, this prevents the user from updating Guix. Instead one can start a Guix shell as follows:

guix shell guix-build-coordinator

Running

$ guix-build-coordinator agent list

shows nothing, so the next step is to set up a build agent, which requires some preparations on the machine where the build server is running. Executing

$ guix-build-coordinator agent new
e092df28-3418-4f94-b7f0-a214b03291ee

creates and prints a new random version 4 UUID and stores it in the agents table in its internal database. The next step is to set up authentication; for a small number of build agents, a password file is a suitable approach, otherwise an authentication token, which may be shared among several agents, can be used. So we create a password for the agent by running

$ guix-build-coordinator agent e092df28-3418-4f94-b7f0-a214b03291ee password new
new password: IUwGrsEklfu0kVf_QquCOdzfa6-P52qPlcwBd5YB

which again prints the password and saves it into the coordinator database.

To simplify things for the human brain, we can give the agent a name; since there is not yet a command line argument for this, we take it as a pretense to look more closely at the SQLite database structure. So install the sqlite package into the root profile, launch sqlite3 on the coordinator database, and run the following commands:

# sqlite3 /var/lib/guix-build-coordinator/guix_build_coordinator.db
sqlite> .tables
agent_passwords
agent_tags
agents
…
builds
…
tags
…
sqlite> select * from agent_passwords;
1|e092df28-3418-4f94-b7f0-a214b03291ee|IUwGrsEklfu0kVf_QquCOdzfa6-P52qPlcwBd5YB|2025-08-13 00:00:00
sqlite> .schema agents
CREATE TABLE agents (
       id TEXT PRIMARY KEY,
       description TEXT
, name TEXT, active NOT NULL DEFAULT 1);

There are quite a few tables, but for now we are mainly interested in those related to the agents. We can recover the password in case we forgot to write it down (alternatively we could run guix-build-coordinator agent e092df28-3418-4f94-b7f0-a214b03291ee password), and we see that agents can have a name and a description, and be active or not. So still in SQLite run

sqlite> update agents set name='bedok', description='TBFG agent' where id='e092df28-3418-4f94-b7f0-a214b03291ee';
sqlite> select * from agents;
e092df28-3418-4f94-b7f0-a214b03291ee|TBFG agent|bedok|1

Alternatively, to check that everything has gone well, run (not necessarily as root anymore):

$ guix-build-coordinator agent list
e092df28-3418-4f94-b7f0-a214b03291ee: bedok
  description:
  TBFG agent  active?: true
  0 allocated builds:
  requested systems:
  tags:

Now it is finally time to really set up the build agent! On the machine where it is supposed to run (in our case, this is bedok again, but it could be an arbitrary machine somewhere on the Internet, since all communication would take place over https), we need to handle the password. As usual in Guix, secrets are not saved in configuration that is publicly visible in the store, but rather as separate state; so as root create a file /etc/guix-build-coordinator/agent-bedok-passwd containing the password IUwGrsEklfu0kVf_QquCOdzfa6-P52qPlcwBd5YB created above by the coordinator. Then add the following snippet to the server part of the operating system configuration:

(service guix-build-coordinator-agent-service-type
  (guix-build-coordinator-agent-configuration
    (authentication
      (guix-build-coordinator-agent-password-file-auth
        (uuid "e092df28-3418-4f94-b7f0-a214b03291ee")
        (password-file
          "/etc/guix-build-coordinator/agent-bedok-passwd")))
    (derivation-substitute-urls
      '("https://data.guix.gnu.org"))
    (non-derivation-substitute-urls
      '("https://bordeaux.guix.gnu.org"))
    (systems '("x86_64-linux" "i686-linux"))
    (max-parallel-builds 4)
    (max-parallel-uploads 2)
    (max-1min-load-average 6)))

and reconfigure the machine. Concerning the different parameters, see the documentation. For authentication, we need to provide our uuid and the location of the password-file. Since the agent runs on the same machine as the coordinator, I kept the coordinator field at its default "http://localhost:8745"; otherwise localhost needs to be replaced by the host name of the coordinator, and the protocol should be set to https. The derivation-substitute-urls field has no default; we will discuss it in the next section. The non-derivation-substitute-urls field also needs to be set to avoid compiling each and every package input locally; here I chose to only use the bordeaux build farm, but one could add "https://ci.guix.gnu.org" to also fetch packages from berlin. If the system field is not set, then only packages for the system on which the agent is running (most likely x86_64-linux) are handled; here it is useful to add i686-linux. Or on an ARM machine, the combo '("aarch64-linux" "armhf-linux") makes sense. The numerical parameters can be left at their defaults; here I am trying to limit the load on my four core processor.

If all goes well, we should see lines in the agent logfile /var/log/guix-build-coordinator/agent.log looking like

2025-08-13 00:00:00 (INFO ): starting agent e092df28-3418-4f94-b7f0-a214b03291ee
2025-08-13 00:00:00 (INFO ): connecting to coordinator http://localhost:8745
2025-08-13 00:00:00 (INFO ): running 0 threads, currently allocated 0 builds
2025-08-13 00:00:00 (INFO ): starting 0 new builds

and running

$ guix-build-coordinator agent list

on the coordinator machine again should now print two entries beneath requested systems.

As a last step, get back to the /etc/guix-build-coordinator/agent-bedok-passwd file, which is probably world-readable. Starting the build agent has created a user guix-build-coordinator-agent, and I would recommend to have the file be owned by that user, and remove permissions from all other users.

Data service

This is the elephant in the room, almost literally. When starting my project of the TBFG, I had intended to set up the full stack of software needed to run the bordeaux build farm, and the Guix Data Service is a very important part of it. But the code base is massive (more than 30000 lines of Scheme code at the time of writing), and also the required ressources are massive: The service spends its time polling the git repository of Guix, compiling the sources and computing all the derivations for all the packages, which are then stored in a database. Rinse and repeat for the next commit. (In reality, the data service does even more, in particular it also queries build servers and stores information about builds; but the above functionality is everything we need for the TBFG.) As can be seen, not even the official data service, running continuously on a powerful server, manages to do that for all commits: Some of them are marked as green, others, the grey ones, are skipped, in particular at times of high commit activity.

So I have decided to rely on the central Guix data service by configuring the corresponding derivation-substitute-urls field of the build coordinator as '(https://data.guix.gnu.org). All information obtained by clicking through the data service website can also be obtained as JSON through its REST API, which we will use in our scripts to determine the derivations of science team packages to be built.

For this to work, we also need the Guix daemon to accept the signing key of the data service; so the services field of the operating system declaration should look like this:

(services
  (append
    (modify-services %base-services
      (guix-service-type config =>
        (guix-configuration
          (substitute-urls '("https://bordeaux.guix.gnu.org"))
          (authorized-keys
            (list
              (local-file "keys/guix/bordeaux.guix.gnu.org-export.pub")
              (local-file "keys/guix/data.guix.gnu.org.pub")))
          (max-silent-time (* 24 3600))
          (timeout (* 48 3600)))))
    (list
      (service guix-build-coordinator-service-type
        (guix-build-coordinator-configuration))
      …)))

where the key files are copy-pasted into the local keys/guix subdirectory from the corresponding place in the guix/maintenance git repository.

BFFE – Build Farm Front End

We could start submitting build jobs now, but to visualise what is happening, we need another service, bffe. The build farm frontend actually serves two purposes: On one hand on the bordeaux build farm, it submits build jobs for the master branch to ensure continuous substitute availability (and a different service, qa-frontpage, submits build jobs for testing branches and pull requests to the same build coordinator instance). On the other hand, it provides a web server that connects to the build coordinator and shows information about its status. We will only need the second functionality. For this, add the following snippet to the service configuration of the TBFG machine:

(service bffe-service-type
  (bffe-configuration
    (arguments
      #~(list
        #:web-server-args
          '(#:event-source "http://localhost:8746"
            #:controller-args (#:title "Science team build farm"))))))

We use the #:web-server-args argument and provide as event-source the local build coordinator instance, which communicates with clients on port 8746 (while communication with the agents runs on port 8745 as seen above). For more details on the optional #:build argument, see the documentation in the Guix manual or the source code.

Reconfigure the system and open the BFFE website, either at http://localhost:8767 locally on the TBFG machine, or at http://192.168.1.80:8767 from another machine in your local network, where you have to adapt the IP address to your situation. The result is a rather empty web page, but it should at least show the title we have chosen. For our purposes, we are interested in the page obtained by appending /activity to the URL. This shows a box Recent activity, which is rightfully empty; and a list of agents per architecture and their current occupation, which should also be void, except possibly for the percentage giving the CPU load in case the agent also has other business. Clicking on the name of an agent sends us to yet another page with more details (among which the description we provided previously) .

Submitting a build

After all these preparations, we can finally submit our first build job! For this, keep the /activity page open, and run

$ DRV=`guix build hello --derivations`
$ guix-build-coordinator build --derivation-substitute-urls=https://data.guix.gnu.org $DRV
build submitted as 7bdd2249-1214-431a-aa61-de4d907f1b32

Providing a URL from which to fetch derivations is necessary because the build coordinator receives only the store path of the derivation and not the actual file; the same then recursively holds for inputs referenced in the submitted derivation. This could be a global parameter of the guix-build-coordinator-configuration, but currently needs to be specified by hand and can thus vary over time or depending on the build.

If all goes well, a UUID for the build is printed in the terminal, and three lines appear on the BFFE web page in the Recent activity box: Build submitted, Build started and Build succeeded. The /var/log/guix-build-coordinator/coordinator.log and /var/log/guix-build-coordinator/agent.log files should also contain matching information. And you can go to the http://192.168.1.80:8767/build/7bdd2249-1214-431a-aa61-de4d907f1b32 URL to see more detailed information on the build, with potentially a link to the build log.

This link unfortunately does not work out of the box, and some more configuration is required to make the build logs available.

Nginx for the build logs

One possibility for accessing the build logs is by directly looking them up in the place where they are stored by the build coordinator, the directory /var/lib/guix-build-coordinator/build-logs/. Each build corresponds to a subdirectory named after its UUID and containing the file log.gz.

Alternatively, we can imitate the behaviour of the bordeaux build farm and set up a separate nginx web server using the following service snippet:

(service nginx-service-type
  (nginx-configuration
    (server-blocks
      (list
	(nginx-server-configuration
	  (listen '("80" "[::]:80"))
	  (locations
	    (list
	      (nginx-location-configuration
		(uri "~ \"\\/build\\/([a-z0-9-]{36})/log$\"")
                (body '("alias /var/lib/guix-build-coordinator/build-logs/$1/log;"
                        "add_header Content-Type 'text/plain; charset=UTF-8';"
                        "gzip_static always;"
                        "gunzip on;")))
              (nginx-location-configuration
                (uri "/")
                (body '("proxy_pass http://localhost:8767;"
                        "proxy_http_version 1.1;"
                        "proxy_set_header Connection \"\";")))
              (nginx-location-configuration
                (uri "/events")
                (body '("proxy_pass http://localhost:8767;"
                        "proxy_http_version 1.1;"
                        "proxy_buffering off;"
                        "proxy_set_header Connection \"\";"))))))))))

The first nginx-location-configuration serves the build logs, while the other two are reverse proxies towards the pages provided by BFFE at port 8767. If the activity page is now accessed through the nginx web server at the standard port through the URL http://192.168.1.80/activity, it presents links to builds and their log files, which can be clicked on, and the uncompressed log files are shown directly in the browser.

Outlook

This was a lot of work for setting up the necessary services! But hopefully it was also a good occasion to understand how the different components interact. In the next installment we will start writing our own scripts to communicate with these services; in particular we will work with the data service to retrieve information about the packages we are interested in.