enge@inria

Autonomie et souveraineté numerique

2026-04-14T00:00:00Z

Le texte suivant a été écrit en collaboration avec Michaël Ferrec d'Inspeere. Il a été inspiré par des discussions au sein du groupe de travail sur la souveraineté numérique d'ENTER, le pôle de compétitivité Excellence Numérique au service des Transitions Environnementales et Responsables en Nouvelle Aquitaine. Il tente justement de proposer une définition du terme sous-jacent aux travaux du groupe.

Définition et terminologie

La notion de souveraineté s'applique strictement a un état ou un territoire. Pour une organisation (entreprise, collectivité, administration) ou un individu, il convient de parler plutôt d'autonomie: la capacité de faire des choix sans interférence extérieure et de les faire évoluer dans le temps. La résilience face aux chocs extérieurs en fait partie. Dans les deux cas, il s'agit d'un axe, pas d'un état binaire. Cet axe va de la dependance totale à la souveraineté idéale, théorique et inatteignable. Ce qui définit la position d'un acteur sur cet axe, c'est sa capacité à identifier, mesurer et gérer ses risques de dépendance. L'autonomie n'est pas une fin en soi: elle s'évalue en regard des objectifs et des moyens disponibles.

Principe fondateur: On ne peut pas déléguer son independance. Confier sa souveraineté à un tiers, c'est y renoncer par définition, quelle que soit l'etiquette apposée sur le service.

Les enjeux selon l'échelle

Au niveau de l'organisation

Continuité opérationnelle: un fournisseur racheté ou soumis à une juridiction étrangere peut interrompre l'activité du jour au lendemain.
Risque juridique: l'hébergement par un acteur soumis au Cloud Act ou FISA expose à des accès non maîtrisés.
Dépendance économique: la captivité technique réduit la capacité à négocier et à migrer, et génère une hausse structurelle des couts.

Au niveau du territoire

À cette échelle, les enjeux sont géostratégiques. La dépendance technologique n'est plus seulement un risque opérationnel: c'est une vulnérabilite nationale en cas de crise.

Autonomie décisionnelle: en situation de crise géopolitique ou militaire, une nation dépendante d'acteurs étrangers sur ses infrastructures critiques ne peut plus décider librement.
Rapports de force: la dépendance technologique est devenue un levier de pression diplomatique (sanctions, restrictions d'exportation, décisions unilatérales).
Développement industriel: externaliser massivement vers des acteurs extra-européens revient a financer des filières étrangeres. La commande publique orientée filière est le principal levier correctif.

Les dimensions d'analyse

Sept dimensions permettent de mesurer la position sur l'axe. Elles sont cumulatives et interdépendantes.

Logiciel

L'autonomie logicielle est étroitement lieé aux libertes du logiciel libre: liberté d'exécuter, d'adapter, de redistribuer.

Je suis autonome si j'utilise un logiciel libre développé en interne.
Je suis moins autonome si j'utilise un logiciel libre développé par d'autres, ou un logiciel propriétaire développé en interne.
Je suis moins autonome si j'utilise un logiciel propriétaire développé par un tiers.

Données

L'autonomie se mesure par l'utilisation de formats ouverts et interopérables. La nature des logiciels disponibles pour traiter les données entre egalement en jeu.

Matériel

L'essentiel du materiel numérique etant concu et fabriqué à l'étranger, la question se rapproche de celle de la souveraineté et pose le probleme de la résilience des chaînes d'approvisionnement.

Je suis autonome si j'emploie du matériel de ma propre conception, fabriqué en Europe.
Je suis moins autonome si j'emploie du matériel standard disponible chez un grand nombre de fournisseurs en compétition.
Je suis moins autonome si j'ai besoin de matériel dedié, fabriqué en petites series, ou concu comme boite noire non modifiable.

Infrastructure et hébergement

Il s'agit de l'endroit ou logiciels et données sont mis ensemble, pour un usage interne ou externe.

Je suis autonome si j'héberge mes logiciels et données sur mon propre matériel (on premise).
Je suis moins autonome si j'héberge chez quelqu'un d'autre (datacentre tiers).
Je suis moins autonome si mes données sont hébergees et traitées par des logiciels tiers en ligne (cloud).

Juridique

Le cadre juridique peut limiter l'autonomie ou la garantir. On est plus autonome dans un état de droit liberal et dans sa propre juridiction.

Je suis autonome si j'opère mes activités numériques en Europe.
Je suis moins autonome si j'opère dans un état de droit hors Europe.
Je suis moins autonome si j'opère dans un régime juridique opaque ou instable.

Connaissances

L'autonomie augmente si l'on choisit des solutions bien documentées pour lesquelles il existe un grand nombre de personnes formées.

Je suis autonome si j'emploie des solutions largement répandues, enseignées et bien documentées.
Je suis moins autonome si j'emploie des solutions ilôt pour lesquelles il y a un monopole sur la connaissance.

Économique

La finitude des ressources limite l'autonomie. L'absence de concurrence peut entraîner une dépendance vis-à-vis d'un fournisseur et rendre des choix difficilement réversibles.

Je suis autonome si je choisis des solutions compatibles avec mes moyens, sur un marché concurrentiel.
Je suis moins autonome si je dépends d'un fournisseur unique qui à le monopole sur la solution.

La limite de la délégation

Une organisation ne peut pas être spécialiste de toute son infrastructure. Il est légitime de confier l'exploitation à un tiers. La question est de savoir à quel moment cette delegation devient une perte de maîtrise. La distinction fondamentale: on peut déléguer l'exploitation, pas la maîtrise. La question n'est pas "est-ce que je gère moi-même?" mais "est-ce que je peux reprendre la main?"

Quatre critères permettent d'évaluer si une délégation est saine ou si elle constitue une perte de contrôle :

Réversibilité: puis-je partir et récuperer mes données dans des formats exploitables?
Criticité: ce composant est-il central à mon activité ou périphérique?
Transparence: sais-je ce qui se passe sur mes données? Un audit est-il possible?
Concurrence: existe-t-il des alternatives crédibles, ou suis-je face à un monopole de fait?

Conclusion

La souveraineté parfaite n'existe pas, ni pour une organisation ni pour un territoire. L'objectif n'est pas l'indépendance absolue mais la maîtrise consciente de ses dépendances: connaitre sa position sur l'axe, comprendre les risques associés, et progresser méthodiquement sur les dimensions les plus critiques. L'absence de politique assumée à ce sujet est elle-même un choix, dont le coût ne devient pleinement visible qu'en temps de crise, quand les rapports de force s'exercent et que la dépendance se transforme en vulnerabilité.

Primality record with ECPP – 109297 digits

2026-02-11T00:00:00Z

ECPP

Heuristically, the ECPP algorithm (more precisely, its FastECPP variant) is the fastest algorithm for certifying generic primes (as opposed to merely providing a one-sided test that can only certify that a prime is composite, or find out that with high probability it is prime without being certain). As an additional advantage, it provides a certificate that can be checked in polynomial time with a lower exponent than the certificate creation. My CM software has been used for most of the current record computations (it corresponds to all the user codes starting with the letter "E", the following number encodes the set of participants to the computation). My previous record for a repunit with 86453 digits, that is, the number (10⁸⁶⁴⁵³ - 1)/9 consisting of 86453 successive digits 1, has been described in some detail in the publications FastECPP over MPI at the International Congress on Mathematical Software 2024, and it has been publicised in the Dutch newspaper NRC. This little note gives an update for the most recent record, the repunit with 109297 digits, an endeavour undertaken jointly with Paul Underwood using my CM software.

The two latest ECPP records

Let me first summarise a few numbers about the previous proof of the 86453 digit repunit. The first phase, which determines the parameters of a sequence of auxiliary elliptic curves, has been run on the PlaFRIM cluster, repeatedly with checkpointing for three days in a row, on a varying number of 759 to 2639 cores, depending on how busy the cluster was. In total, it has taken 383 CPU years and less than 4 months in real time, determining successively the parameters of 2979 elliptic curves. The CM software communicates via MPI over TCP, which enables it to run on a heterogeneous cluster of machines; it has mainly used the standard nodes of the plafrim cluster, in particular the zonda and diablo machines with 32-core AMD Zen2 Rome EPYC 7452 @ 2.35 GHz processors and other diablo machines with 64-core AMD Zen2 Rome EPYC 7702 @ 2 GHz and 64-core AMD Zen3 Milan EPYC 7763 @ 2.45 processors. The second phase, which computes the elliptic curves using the complex multiplication method, has been run on brise, a single machine with 96 cores Intel Xeon E7-8890 v4 @ 2.2GHz, which are older and slower, but the machine has 1TB of RAM, which is important during this part of the algorithm. Being slower, it is also less popular, so that I could use it for more than three days in a row. (Thanks to the administrators for being flexible and helpful!) The second phase has taken about 25 CPU years and also about 4 months of real time. I have verified the certificate with PARI/GP (it is a good idea to use different software for the verification) again on brise with its 96 cores, which took 190 CPU days, or almost two days of real time.

For the new record of a 109297 digit number, Paul Underwood has carried out the first phase on a single machine with 64 cores AMD Ryzen Threadripper 3990X Zen2 @ 2.9GHz, which took 87 CPU years and 21 months of real time, for a certificate with 6847 steps. The second phase has been carried out by me on the same machine with 96 cores as the previous record, which took 133 CPU years and also 21 months of real time. We ran the two phases mostly in parallel, so that the whole effort took about two years.

Comparing the first phases

As a first observation, it is remarkable that the first phase on a larger number took actually less CPU time! This is due to the fact that this phase is not embarrassingly parallel, unlike many computations in number theory; and as noticed in the accompanying publication, the 86453 digit record was probably over-parallelised. To measure things, we need to consider the size of the problem, which here is given by the number L of digits of the number to be proved prime. The proportion of sizes between the records is 109297/86453, or approximately 1.26. Since both phases of the algorithm have a complexity of O~(L⁴), one would expect the new record to take about 1.26⁴ or 2.6 times as long as the old one. But in reality, the CPU time was lower by a factor of about 4.4; so between the actual performance and the prediction by the asymptotic complexity there is a factor of about 11.

Part of this can be explained by different single-core performance. Paul's machine has a higher clock rate, and he has modified the CM code to link with gwnum, a library for integer arithmetic using a floating-point FFT, which may lead to errors depending on the choice of parameters (but these errors can be found by doing an even faster test modulo a one-word prime, so they have no impact on the correctness of the final computation; and in any case, we end up with a certificate that is checked independently). So the computations are faster than with GMP, in experiments by a factor of about 1.6 for the size of numbers under consideration. Unfortunately, gwnum is not free software, so not taken into account for the CM development.

Concerning parallelism, during the first phase there is a test whether a number consists of a smooth part multiplied by a prime, where smooth means that the number completely factors into primes less than some bound B. Each core covers a range of about 2²⁹, so that B is about 2²⁹ multiplied by the number of cores. Thus with more cores, on average a larger smooth part is factored out. The next step continues with the new, smaller prime number, so that in fact the size of the smooth part corresponds to the number of digits gained in one step of the algorithm. Unfortunately this is not proportional to B, but only to log B. Dividing the size of the numbers by the length of the certificate, we see that we gained on average 96 bits per step in the old record and only 53 bits per step in the new record. If we assume that the first record was carried out with about 1500 cores (where in reality the number varied over the course of the computation) and the second one with 64 cores, then the ratio between the gains per step should be about (log (2²⁹) · 1500) / (log (2²⁹) · 64) or 1.13, which on one hand is a small gain for employing 20 times as many cores; and on the other hand does not match the observed factor of 96/53 or 1.8. However, there is another impact of using more cores: It increases the number of suitable curves found at any given step, of which only one can be kept. When a round of computations returns several candidates, the CM software chooses the one with the greatest gain in bits. With only 64 cores, there will often be no choice; with 1500 one can often choose between several ones. So to make a long explanation short: Using more cores leads to shorter certificates, which are found in less wallclock time, but at the expense of an increase in total CPU time.

Comparing the second phases

In the second phase, which for both records has been carried out on the same machine with the same multiprecision library so that the comparison becomes easier, the CPU time has indeed increased compared to the previous record, but by a factor of 133/25 or 5.3 instead of the theoretically predicted (109297/86453)⁴ or 2.6, which may be surprising at first. But we can do a more precise estimate, which takes the certificate length into account: The power 4 comes from a power 3 related to the arithmetic (root finding of the class polynomial modulo the prime to be certified) plus one power from the length of the certificate. Since the certificate for the new record has overproportionally more steps due to less parallelisation as seen above, we should use the better estimate (109297/86453)³ · (6847/2979) or 4.6, which essentially explains the running time. So with less parallelisation in the first phase, there is also a price to pay in the second phase.

Conclusion

The certificate files for the 109297 digit repunit are available from the CM ECPP page.

As is often the case with complex algorithms that depend on the careful choice of several parameters, it is not easy to predict their running times on inputs of varying size. The above musings try to explain our running time observations, but cannot claim to be the absolute truth. One thing is certain: For now, proving the next prime candidate for a repunit is out of reach. OEIS A004023 lists it as the repunit with 270343 digits; using the asymptotic complexity to obtain an estimated running time, we find that it will take (270343/109297)⁴ or 37 times as long as the current record. Otherwise said, instead of 21 months, it should take 66 years! So if you want to prove a new record prime, make sure to choose it from a different source.

Tiny Build Farm for Guix, part 2

2025-10-24T00:00:00Z

Building science packages

In our efforts to create a Tiny Build Farm for Guix, that is supposed to report on the status of the packages assigned to the science team, so far we have seen how to set up the required infrastructure. On a dedicated machine with Guix as its operating system, we have added several Shepherd services: the Guix Build Coordinator together with a build agent; and the web server part of the BFFE, which enables us to follow the activity of the builders. For performance reasons, we have renounced at installing an instance of the Guix Data Service, and opt instead for talking to the instance operated by the Guix project at https://data.guix.gnu.org/, which continually evaluates the Guix master branch and creates derivations for all packages in the distribution. The next step is to explore how to programmatically talk to the remote data server from a Guile script, how to extract derivations we are interested in, and how to submit them for building to our instance of the build coordinator.

Getting information from the data service

We need to install the two packages guix-data-service and (for later use) guix-build-coordinator on the TBFG machine, which contain Guile libraries with the necessary functionality.

⚠ If installed into a user profile, both packages pull in the guix package as a propagated input, which prevents the user from updating it through guix pull. It is thus recommended to run

guix shell guile-next guix-data-service guix-build-coordinator

instead. At the time of writing, the guile package in Guix is at version 3.0.9, while the data service library requires guile-next, which is at version 3.0.10.

Let us open a Guile REPL and execute the following code (to ease copy-pasting, I omit the prompt of the REPL; lines starting with a $ sign and a number correspond to results).

$ guile
(use-modules (guix-data-service client))
(define my-data-service "https://data.guix.gnu.org/")

(define json
  (guix-data-service-request my-data-service
                             "repository/1/branch/master.json"))
json
$1 = (("revisions" . #((("data_available" . #f) ("commit-hash" . "cb47639a8081e8e2d651ad1612bbd1e482766469") …

The call to guix-data-service-request is equivalent to opening the URL https://data.guix.gnu.org/repository/1/branch/master.json, which executes the same query as the URL https://data.guix.gnu.org/repository/1/branch/master without the .json at the end, but it returns the result in JSON format. Moreover, the function call transforms the JSON into a Guile data structure through the guile-json library; in particular, JSON arrays become Guile vectors and JSON objects become Guile association lists, or alists for short (these are lists of key-value pairs, so brace yourself for lots of parentheses in a row). Thus parsing the result and extracting the information we are interested in amounts to unwrapping these successive layers; in true Scheme/Lisp style we will also usually transform the vectors into lists using the vector->list function. The JSON we asked for is an object with a unique field revisions, which contains an array of revisions, that is, git commits on the master branch; every revision is an object with the three fields date, commit-hash (these are strings) and data_available, a boolean indicating whether the data service has computed the derivations for this commit or not (which corresponds to the green or grey badges on the website). This structure can be derived by looking at and playing with the variables in the REPL, or probably more conveniently by opening the corresponding URL in a web browser, which should show the JSON in a special mode. We can now write a small function (or maybe two even smaller functions) that query the data service and return a list of revisions for which the data service has computed the derivations:

(define (data-available? revision)
  ;; Given a REVISION, check whether it has been treated by the
  ;; data service.
  (assoc-ref revision "data_available"))

(define (get-revisions data-service)
  ;; Query DATA-SERVICE for the list of revisions it has successfully
  ;; treated in the master branch.
  (filter data-available?
    (vector->list
      (assoc-ref
        (guix-data-service-request data-service
          "repository/1/branch/master.json")
        "revisions"))))

(define revisions (get-revisions my-data-service))
revisions
$2 = ((("data_available" . #t) ("commit-hash" . " …

In the following, we will work with revisions in this form, although mainly the commit hashes are of interest. We could print them as follows:

(define commits
  (map (lambda (revision)
         (assoc-ref revision "commit-hash"))
       revisions))
commits
$3 = ("b966f4007c8492ad89eedf32dd91b3352dba594e" "8a1f56cf8710fc142a2f8ef2e52be82e8aa9f53e" …
(length commits)
$4 = 46
(define commit (car commits))
commit
$5 = b966f4007c8492ad89eedf32dd91b3352dba594e

By default the data service returns 100 revisions (including those for which no data is available), which will be amply enough for our purposes.

The next step is to obtain the derivations for a given revision, say the newest one with data available. Again this is most easily reverse-engineered from the web interface of the data service: Click on the latest revision with a green badge, then on View package derivations; this shows how the URL is to be formed. Since we need all derivations, we also have to tick the All results checkbox; on the other hand, we may limit to one architecture, say x86_64-linux as System, and not consider cross-compilation by choosing (no target) for Target. These choices add GET parameters to the query, which can be passed as an alist for the optional third parameter of guix-data-service-request. Again adding .json to the URL (in front of the ?) shows the structure of the resulting JSON. It is then easy to end up with the following function; notice the use of the quasiquote ` and the unquote ,:

(define (get-derivations data-service commit system)
  ;; Query DATA-SERVICE for the list of derivations for the given COMMIT
  ;; and SYSTEM.
  (map
    (lambda (p)
      (assoc-ref p "derivation"))
    (vector->list
      (assoc-ref
        (guix-data-service-request data-service
          (string-append "revision/" commit "/package-derivations.json")
          `((system . ,system) (target . "none") (all_results . "on")))
        "derivations"))))

(define derivations
  (get-derivations my-data-service commit "x86_64-linux"))
(length derivations)
$6 = 29531
(car derivations)
$7 = "/gnu/store/000lxmn2d17bv2v6znvf6z5vi7ndy8q4-r-janeaustenr-1.0.0.drv"

So the derivations are simply strings pointing to files in the store (of the data service, so far they are not yet available on the TBFG machine).

Filtering out team packages

29000 derivations are more than our poor tiny machine can handle; the next step is to filter out those that correspond to packages in the science team. The team is responsible for certain package modules (or equivalently, for .scm files in the gnu/packages/ directory); which ones can be seen in the file CODEOWNERS checked into the Guix git repository, itself derived from etc/teams.scm. As it does not change very often, for simplicity we may determine the list of modules by hand, which may require us to resolve regular expressions (here: fortran(-.+|)) into lists of actually present modules; here we end up with the following:

(define my-locations
  '("algebra" "astronomy" "chemistry" "fortran-check" "fortran-xyz"
  "geo" "graph" "lean" "maths" "medical" "sagemath" "statistics"))

When starting the project, I had hoped to extract the interesting packages directly from the (strings representing) derivations, given a fixed list of package names. But it is a truth universally acknowledged that a programmer never has the singularly good fortune of such simplicity, whatever their feelings or views when first entering the neighbourhood of a problem. Here two reasons speak against it: First of all, the packages of a team may change over time as packages are added, removed or moved to a different module. More immediately, though, only the combination of package name and version can be easily recovered from the derivation by removing a fixed prefix, the hash and a fixed suffix, using the following function:

(define (derivation->name+version derivation)
  ;; Given a DERIVATION (by a string of the form "/gnu/store/..."),
  ;; return the part of it that encodes the name and the version
  ;; of the underlying package.
  (string-drop (basename derivation ".drv") 33))

Thus /gnu/store/000lxmn2d17bv2v6znvf6z5vi7ndy8q4-r-janeaustenr-1.0.0.drv becomes r-janeaustenr-1.0.0, which is the concatenation of the package name (which is mostly fixed over different revisions) and the package version (which usually increases over time) with a hyphen in-between. More often than not it is possible to guess the two components: Here they are r-janeaustenr and 1.0.0. Package names often contain hyphens (like here, they serve to separate a language part, r, and the upstream name, janeaustenr, see the Guix naming conventions); this could be handled by splitting at the last hyphen, but versions may also contain hyphens. Both can contain alphabetic and numeric components. Thus it would be quite possible that the above derivation is for the flourishingly named version janeaustenr-1.0.0 of the r package.

So we need more code to extract the desired information. Luckily the data service knows about the packages in a revision, with their names and their versions in different fields; and also about their locations, that is, the files in which they are defined.

(define (get-packages data-service commit)
  ;; Query DATA-SERVICE for the list of packages for the given COMMIT.
  (vector->list
    (assoc-ref
      (guix-data-service-request data-service
        (string-append "revision/" commit "/packages.json")
        `((field . "version") (field . "location") (all_results . "on")))
      "packages")))

(define packages (get-packages my-data-service commit))
(car packages)
$8 = (("location" ("column" . 2) ("line" . 8273) ("file" . "gnu/packages/games.scm")) ("version" . "0.27.1") ("name" . "0ad"))

It is now enough to compare the file name with our list of locations to extract the packages we are interested in.

(define (location-package? package locations)
  ;; Check whether the PACKAGE comes from the list of LOCATIONS.
  (let* ((file (assoc-ref (assoc-ref package "location") "file"))
         (module (basename file ".scm")))
        (member module locations)))

(use-modules (srfi srfi-26))
(define (packages-name-version data-service commit locations)
  ;; Query DATA-SERVICE for a list of packages for the given COMMIT
  ;; that come from the list of LOCATIONS. Return a list of two-element
  ;; lists with the names and versions of these packages.
  (map
    (lambda (package)
      (list (assoc-ref package "name") (assoc-ref package "version")))
    (filter
      (cut location-package? <> locations)
      (get-packages data-service commit))))

(define team-name-versions
  (packages-name-version my-data-service commit my-locations))
(car team-name-versions)
$9 = ("4ti2" "1.6.12")

Finally we just need to compare the extracted team package names and their versions with the derivations. Unfortunately this can be quite costly; the following code presents a somewhat optimised solution with memory usage linear in the result, but a quadratic number of comparisons (thanks to Liliana Prikler for suggesting the use of filter-map to me):

(use-modules (srfi srfi-1))
(define (special-cartesian-product X Y)
  ;; Let X and Y be lists of two element lists of the form (x z) and (y z),
  ;; respectively. Return a list of all the (x y) such that there is an
  ;; element z with (x z) in X and (y z) in Y.
  (fold cons '()
        (filter-map (lambda (xz)
                      (let ((yz (find (lambda (yz)
                                        (equal? (cadr xz) (cadr yz)))
                                      Y)))
                        (if yz
                            (list (car xz) (car yz))
                            #f)))
                    X)))

(define (team-derivations data-service commit system locations)
  ;; Query DATA-SERVICE for the list of derivations for the given COMMIT
  ;; and SYSTEM, filtered by the LOCATIONS of the packages.
  ;; To memorise the computed information, return a list of two element
  ;; lists, each containing a derivation and the corresponding name.
  (let* ((derivations (get-derivations data-service commit system))
         (X (map
              (lambda (d)
                (list d (derivation->name+version d)))
            derivations))
         (name-versions
           (packages-name-version data-service commit locations))
         (Y (map
              (lambda (nv)
                (list (car nv) (string-append (car nv) "-" (cadr nv))))
            name-versions)))
    (special-cartesian-product X Y)))

(define (sort-derivation-names derivation-names)
  ;; Just for the fun of it, sort DERIVATION-NAMES, a list of two element
  ;; lists containing derivations and their names, by names.
  (sort derivation-names
        (lambda (x y)
          (string<? (cadr x) (cadr y)))))

(define good-derivation-names
  (sort-derivation-names
    (team-derivations my-data-service commit "x86_64-linux" my-locations)))
(define derivation-name
        (find (lambda (dn)
                (equal? (cadr dn) "lrslib"))
              good-derivation-names))
derivation-name
$10 = ("/gnu/store/3pxq1g2java4f8nwfq7n98qjvhkr1b34-lrslib-7.2.drv" "lrslib")

Strictly speaking, the function team-derivations is not correct; if there were simultaneously a derivation for the package r-jauneaustenr at version 1.0.0 and a derivation for the package r at version jauneaustenr-1.0.0, then either both or none of them would match, while it is possible that only one of the packages is covered by the science team, a situation not yet encountered; at worst, we would capture one too many derivations. For testing purposes during the development of the TBFG, we additionally check whether the name equals lrslib; in this way only one derivation is returned (while at the time of writing there are more than 700 packages covered by the science team). Moreover the package in question is a self-contained C program (without any inputs), which compiles rather quickly.

Submitting builds

Now that we have a list of derivations, we would like to submit them from our Guile script to the build coordinator. This is not very different from the approach seen last time for submitting from the command line. Again it is recommended to open a browser window on the /activity page of the BFFE to see the build coordinator and the agent in action.

(use-modules (guix-build-coordinator client-communication))

(define my-build-coordinator "http://localhost:8746")
(define ignore-if-build-for-derivation-exists? #f)
(define ignore-if-build-for-outputs-exists? #f)
(define ensure-all-related-derivation-outputs-have-builds? #f)
(define priority 0)

(define (submit-build build-coordinator data-service derivation tags)
  ;; Given a DERIVATION (as a string), submit it to BUILD-COORDINATOR
  ;; together with TAGS;
  ;; DATA-SERVICE is passed through and used by the build coordinator to
  ;; obtain the derivation file and further references contained in
  ;; DERIVATION.
  (send-submit-build-request
    build-coordinator derivation (list data-service) 0 priority
    ignore-if-build-for-derivation-exists?
    ignore-if-build-for-outputs-exists?
    ensure-all-related-derivation-outputs-have-builds?
    tags))

(submit-build my-build-coordinator my-data-service (car derivation-name) '())
$11 = (("build-submitted" . "8f8f1cad-fe9c-462c-bc59-3d1f87abf942"))
$12 = #<<response> …

The global variables, which we pass on to the submit-build function, determine the behaviour of the build coordinator. If ignore-if-build-for-derivation-exists? is true, then the build will not be carried out a second time if it was already tried (successfully or not) by the build coordinator before. In production, it will thus be preferable to set it to #t; while still experimenting, we are likely to submit the same derivation several times. Setting the value to #f would also make sense to check that rebuilding the same package works. The variable ignore-if-build-for-outputs-exists? goes a bit further; if set to #t, then the build will not be carried out if a different derivation with the same output was already tried (a very technical distinction; I would recommend to leave it at #f). If ensure-all-related-derivation-outputs-have-builds? is #t, then the build coordinator will recursively submit builds for all the derivations required as inputs to a given derivation. While this sounds reasonable at first, it can go very far, since the coordinator does not look at the store, but at the builds it has handled itself and recorded in its database. This means that the first build submission, when the database is still empty, will entail a complete bootstrap of the Guix distribution. So I would recommend to leave it also at #f. Then the build works as follows: The coordinator sends the derivation to an agent. The agent tries to download all required inputs from a substitute server and if successful, will build only the derivation it is asked to build. Otherwise, it reports back to the coordinator that it has encountered a set-up failure, together with a list of missing inputs. This triggers a hook in the coordinator, and the default hook is to add the missing inputs to the list of outstanding builds, as well as the failed build itself to try it again once the inputs are available. In this way, even if ensure-all-related-derivation-outputs-have-builds? is #f, all really missing inputs will be built recursively, until the build succeeds or a real failure in one of its inputs is encountered.

The submission immediately returns two values, without waiting for the package build to finish. The first return value can be used to link the submitted derivation to the shown UUID of the build, which is a key in the build coordinator database. The second return value is the HTTP response, which we will ignore from now on.

Tags can be added in a parenthesis rich format; the parameter is a list of tags, where each tag is a two element list (not a pair!), in which both elements are pairs. The first one pairs the keyword key to a value, the second one pairs the keyword value to a value (the values are used to construct the URL and can be strings or numbers). So the following would work:

(define tags `(((key . "commit")(value . ,commit))
               ((key . "name")(value . ,(cadr derivation-name)))
               ((key . "build")(value . 2))))
(submit-build my-build-coordinator my-data-service (car derivation-name) tags)
$13 (("build-submitted" . "82a56cac-1e93-4b4a-926f-d8762f919219"))
$14 = #<<response> …

The tags are shown in the activity window and are also recorded in the build coordinator database; as shown here, they can encode arbitrary additional information of a build, such as the commit it comes from, the package name or the submission count for a given derivation.

Code

For ease of use, the code developed in this post is made available, under GPLv3 or later, in a dedicated git repository on Codeberg. More precisely, it is collected in the file tbfg.scm at commit 51eb5c6d45c66d15b7c14340ec3af0732b5b66fd.

Outlook

We have queried the data service and used the resulting information on packages and derivations to submit build jobs to the build coordinator. But so far we have no programmatical access to the build results; we only saw the builds flicker by on the BFFE website. It would be nice to record success or failure, and more generally to keep track of the builds; this will be our next step. Since we do not want to operate a substitute server, but rather follow the state of the packages under the responsibility of the science team, unlike the official build farms we are not necessarily interested in obtaining the build results. These are sent from the build agents to the build coordinator; on the bordeaux build farm the nar herder shovels them to a separate substitute server. For us everything is on the same machine, which will thus contain successfully built packages in its store (at least until the next guix gc run). If desired, these could be made available using guix publish.

Tiny Build Farm for Guix, part 1

2025-08-27T00:00:00Z

Setting up scores of services

One of the oft-cited reasons people give for not switching to Guix is that their favourite software is too outdated, and a look at Repology shows that they are not wrong. Now the number of active committers in the Guix project is amazingly small, and even counting all contributors I am impressed by what these few people actually achieve. Nevertheless I wondered how I could improve the situation at least a little bit for packages I am interested in, that is, for the science team; and the first step is to get an account of what actually builds and what does not. So I decided to set up my own little build farm, limited to the packages in the scope of the science team, using the same technology that powers the bordeaux build farm. I call it the Tiny Build Farm for Guix, or TBFG for short, and this post is the first one in (hopefully) a series of blog posts about the topic; at the time of starting this series, the TBFG does not actually exist yet, so wish me luck.

Motivation

Before trying to solve a technical problem, let me digress a little bit: Is there actually a problem? And if yes, why? As has become my conviction over the years, the really difficult and major problems in a project such as Guix are actually social and not technical. They are rooted in the structure of Guix as a loosely coupled group of volunteers who work on a common goal, mostly in their spare time; but when I speak about a common goal, every volunteer has in fact their own goals, and arriving at a coherent whole is partially due to the internal structuring of the social project, and partially an emerging property of a complex system. Concretely, it happens often that contributors propose a package for a software project they like; if it concerns free software and follows our packaging guidelines, it usually ends up being committed to the software distribution. The original contributor may leave, the package may bitrot and stop being buildable due to changes made in other parts of the distribution. Sometimes people submit a bug report, sometimes committers without interest in the actual software provide a fix, but maybe nobody uses the software anymore, and it happens that over several years nobody notices it is broken. This is problematic since even broken packages use resources on the build farms. And when introducing, say, an update to a library package, it becomes difficult to say whether the failure of a dependency is a new phenomenon due to the change or whether it was already present. So there are good reasons to strive for a distribution that is 100% buildable at all times. And while I alone certainly cannot reach this for the currently more than 28000 packages in Guix, doing it only for the science team, or maybe only the algebra and maths modules, in which I am particularly interested, appears to be a reachable goal.

But is a tiny build farm really needed? The honest answer is “no”, since the information is already out there in the big build farms. For historical reasons, Guix has two of them. One is CI, also called berlin for the location of most of its build machines; it relies on Cuirass, a continuous integration system written purposefully for Guix. It shows the state of the master branch and provides a dashboard from which the desired information could certainly be extracted automatically. On the other hand there is the bordeaux build farm, named after the location of its head node; it runs a suite of continuous integration tools written purposefully for Guix by Christopher Baines. One of its parts is the Guix Data Service, and its REST API with a JSON frontend makes it again possible to extract the information I am interested in – a system that knows about all packages in Guix by definition knows about the packages in the realm of the science team.

So setting up my TBFG is mainly an educational project – I would like to learn how our technology works. But the TBFG can also be used to obtain information about a collection of packages that is not part of Guix proper, or to look at the impact of local changes. Many people and projects have successfully set up Cuirass to manage their local package collection; this is, for instance, the case for the Guix Science and Guix HPC projects through the build server at INRIA Bordeaux. The software behind the bordeaux build farm is more complex and consists of several interconnected Shepherd services, so that it may in fact be less suited for a personal project. However, not least because I host part of that build farm at home, I am more interested in understanding this software stack, which is also less documented. So I am going to build the TBFG on top of this technology. Before diving in, I take the opportunity to thank Christopher Baines for his precious help during a Guix/Nix hackers' meeting; without him, I would not have been able to launch myself into this endeavour.

Build coordinator and agent

As a first step, we need to install the Guix Build Coordinator and one or more build agents. For a really tiny TBFG, I will keep everything on only one machine set aside for the purpose; it is called bedok and is one of the Lenovo Thinkpad X1 Gen9 with a four core 11th Gen Intel Core i7-1165G7 processor running at 2.80GHz graciously donated by Tweag. For the build coordinator, this is trivial; simply add the two lines

(service guix-build-coordinator-service-type
  (guix-build-coordinator-configuration))

to the services configuration of the Guix system declaration and reconfigure the machine. For a start, the default configuration options explained in more detail in the manual are appropriate (things will become more complicated if anything is to be done with the build results, which would require to set the hooks field of the configuration record). See also the documentation in the git repository of the project.

Next we need the guix-build-coordinator package, which provides a command line interface to the coordinator. It could be installed into an arbitrary profile since the security model of the build coordinator is very basic: It is assumed that the coordinator runs on a server of its own, and anybody with access to the machine has control over the service.

⚠ However installing the package into a user profile currently has a big drawback: It propagates the guix package, and the corresponding guix takes precedence over the one in $HOME/.config/guix/current/bin. The latter one is updated by guix pull, but the former one is not; so without a special approach, this prevents the user from updating Guix. Instead one can start a Guix shell as follows:

guix shell guix-build-coordinator

Running

$ guix-build-coordinator agent list

shows nothing, so the next step is to set up a build agent, which requires some preparations on the machine where the build server is running. Executing

$ guix-build-coordinator agent new
e092df28-3418-4f94-b7f0-a214b03291ee

creates and prints a new random version 4 UUID and stores it in the agents table in its internal database. The next step is to set up authentication; for a small number of build agents, a password file is a suitable approach, otherwise an authentication token, which may be shared among several agents, can be used. So we create a password for the agent by running

$ guix-build-coordinator agent e092df28-3418-4f94-b7f0-a214b03291ee password new
new password: IUwGrsEklfu0kVf_QquCOdzfa6-P52qPlcwBd5YB

which again prints the password and saves it into the coordinator database.

To simplify things for the human brain, we can give the agent a name; since there is not yet a command line argument for this, we take it as a pretense to look more closely at the SQLite database structure. So install the sqlite package into the root profile, launch sqlite3 on the coordinator database, and run the following commands:

# sqlite3 /var/lib/guix-build-coordinator/guix_build_coordinator.db
sqlite> .tables
agent_passwords
agent_tags
agents
…
builds
…
tags
…
sqlite> select * from agent_passwords;
1|e092df28-3418-4f94-b7f0-a214b03291ee|IUwGrsEklfu0kVf_QquCOdzfa6-P52qPlcwBd5YB|2025-08-13 00:00:00
sqlite> .schema agents
CREATE TABLE agents (
       id TEXT PRIMARY KEY,
       description TEXT
, name TEXT, active NOT NULL DEFAULT 1);

There are quite a few tables, but for now we are mainly interested in those related to the agents. We can recover the password in case we forgot to write it down (alternatively we could run guix-build-coordinator agent e092df28-3418-4f94-b7f0-a214b03291ee password), and we see that agents can have a name and a description, and be active or not. So still in SQLite run

sqlite> update agents set name='bedok', description='TBFG agent' where id='e092df28-3418-4f94-b7f0-a214b03291ee';
sqlite> select * from agents;
e092df28-3418-4f94-b7f0-a214b03291ee|TBFG agent|bedok|1

Alternatively, to check that everything has gone well, run (not necessarily as root anymore):

$ guix-build-coordinator agent list
e092df28-3418-4f94-b7f0-a214b03291ee: bedok
  description:
  TBFG agent  active?: true
  0 allocated builds:
  requested systems:
  tags:

Now it is finally time to really set up the build agent! On the machine where it is supposed to run (in our case, this is bedok again, but it could be an arbitrary machine somewhere on the Internet, since all communication would take place over https), we need to handle the password. As usual in Guix, secrets are not saved in configuration that is publicly visible in the store, but rather as separate state; so as root create a file /etc/guix-build-coordinator/agent-bedok-passwd containing the password IUwGrsEklfu0kVf_QquCOdzfa6-P52qPlcwBd5YB created above by the coordinator. Then add the following snippet to the server part of the operating system configuration:

(service guix-build-coordinator-agent-service-type
  (guix-build-coordinator-agent-configuration
    (authentication
      (guix-build-coordinator-agent-password-file-auth
        (uuid "e092df28-3418-4f94-b7f0-a214b03291ee")
        (password-file
          "/etc/guix-build-coordinator/agent-bedok-passwd")))
    (derivation-substitute-urls
      '("https://data.guix.gnu.org"))
    (non-derivation-substitute-urls
      '("https://bordeaux.guix.gnu.org"))
    (systems '("x86_64-linux" "i686-linux"))
    (max-parallel-builds 4)
    (max-parallel-uploads 2)
    (max-1min-load-average 6)))

and reconfigure the machine. Concerning the different parameters, see the documentation. For authentication, we need to provide our uuid and the location of the password-file. Since the agent runs on the same machine as the coordinator, I kept the coordinator field at its default "http://localhost:8745"; otherwise localhost needs to be replaced by the host name of the coordinator, and the protocol should be set to https. The derivation-substitute-urls field has no default; we will discuss it in the next section. The non-derivation-substitute-urls field also needs to be set to avoid compiling each and every package input locally; here I chose to only use the bordeaux build farm, but one could add "https://ci.guix.gnu.org" to also fetch packages from berlin. If the system field is not set, then only packages for the system on which the agent is running (most likely x86_64-linux) are handled; here it is useful to add i686-linux. Or on an ARM machine, the combo '("aarch64-linux" "armhf-linux") makes sense. The numerical parameters can be left at their defaults; here I am trying to limit the load on my four core processor.

If all goes well, we should see lines in the agent logfile /var/log/guix-build-coordinator/agent.log looking like

2025-08-13 00:00:00 (INFO ): starting agent e092df28-3418-4f94-b7f0-a214b03291ee
2025-08-13 00:00:00 (INFO ): connecting to coordinator http://localhost:8745
2025-08-13 00:00:00 (INFO ): running 0 threads, currently allocated 0 builds
2025-08-13 00:00:00 (INFO ): starting 0 new builds

and running

$ guix-build-coordinator agent list

on the coordinator machine again should now print two entries beneath requested systems.

As a last step, get back to the /etc/guix-build-coordinator/agent-bedok-passwd file, which is probably world-readable. Starting the build agent has created a user guix-build-coordinator-agent, and I would recommend to have the file be owned by that user, and remove permissions from all other users.

Data service

This is the elephant in the room, almost literally. When starting my project of the TBFG, I had intended to set up the full stack of software needed to run the bordeaux build farm, and the Guix Data Service is a very important part of it. But the code base is massive (more than 30000 lines of Scheme code at the time of writing), and also the required ressources are massive: The service spends its time polling the git repository of Guix, compiling the sources and computing all the derivations for all the packages, which are then stored in a database. Rinse and repeat for the next commit. (In reality, the data service does even more, in particular it also queries build servers and stores information about builds; but the above functionality is everything we need for the TBFG.) As can be seen, not even the official data service, running continuously on a powerful server, manages to do that for all commits: Some of them are marked as green, others, the grey ones, are skipped, in particular at times of high commit activity.

So I have decided to rely on the central Guix data service by configuring the corresponding derivation-substitute-urls field of the build coordinator as '(https://data.guix.gnu.org). All information obtained by clicking through the data service website can also be obtained as JSON through its REST API, which we will use in our scripts to determine the derivations of science team packages to be built.

For this to work, we also need the Guix daemon to accept the signing key of the data service; so the services field of the operating system declaration should look like this:

(services
  (append
    (modify-services %base-services
      (guix-service-type config =>
        (guix-configuration
          (substitute-urls '("https://bordeaux.guix.gnu.org"))
          (authorized-keys
            (list
              (local-file "keys/guix/bordeaux.guix.gnu.org-export.pub")
              (local-file "keys/guix/data.guix.gnu.org.pub")))
          (max-silent-time (* 24 3600))
          (timeout (* 48 3600)))))
    (list
      (service guix-build-coordinator-service-type
        (guix-build-coordinator-configuration))
      …)))

where the key files are copy-pasted into the local keys/guix subdirectory from the corresponding place in the guix/maintenance git repository.

BFFE – Build Farm Front End

We could start submitting build jobs now, but to visualise what is happening, we need another service, bffe. The build farm frontend actually serves two purposes: On one hand on the bordeaux build farm, it submits build jobs for the master branch to ensure continuous substitute availability (and a different service, qa-frontpage, submits build jobs for testing branches and pull requests to the same build coordinator instance). On the other hand, it provides a web server that connects to the build coordinator and shows information about its status. We will only need the second functionality. For this, add the following snippet to the service configuration of the TBFG machine:

(service bffe-service-type
  (bffe-configuration
    (arguments
      #~(list
        #:web-server-args
          '(#:event-source "http://localhost:8746"
            #:controller-args (#:title "Science team build farm"))))))

We use the #:web-server-args argument and provide as event-source the local build coordinator instance, which communicates with clients on port 8746 (while communication with the agents runs on port 8745 as seen above). For more details on the optional #:build argument, see the documentation in the Guix manual or the source code.

Reconfigure the system and open the BFFE website, either at http://localhost:8767 locally on the TBFG machine, or at http://192.168.1.80:8767 from another machine in your local network, where you have to adapt the IP address to your situation. The result is a rather empty web page, but it should at least show the title we have chosen. For our purposes, we are interested in the page obtained by appending /activity to the URL. This shows a box Recent activity, which is rightfully empty; and a list of agents per architecture and their current occupation, which should also be void, except possibly for the percentage giving the CPU load in case the agent also has other business. Clicking on the name of an agent sends us to yet another page with more details (among which the description we provided previously) .

Submitting a build

After all these preparations, we can finally submit our first build job! For this, keep the /activity page open, and run

$ DRV=`guix build hello --derivations`
$ guix-build-coordinator build --derivation-substitute-urls=https://data.guix.gnu.org $DRV
build submitted as 7bdd2249-1214-431a-aa61-de4d907f1b32

Providing a URL from which to fetch derivations is necessary because the build coordinator receives only the store path of the derivation and not the actual file; the same then recursively holds for inputs referenced in the submitted derivation. This could be a global parameter of the guix-build-coordinator-configuration, but currently needs to be specified by hand and can thus vary over time or depending on the build.

If all goes well, a UUID for the build is printed in the terminal, and three lines appear on the BFFE web page in the Recent activity box: Build submitted, Build started and Build succeeded. The /var/log/guix-build-coordinator/coordinator.log and /var/log/guix-build-coordinator/agent.log files should also contain matching information. And you can go to the http://192.168.1.80:8767/build/7bdd2249-1214-431a-aa61-de4d907f1b32 URL to see more detailed information on the build, with potentially a link to the build log.

This link unfortunately does not work out of the box, and some more configuration is required to make the build logs available.

Nginx for the build logs

One possibility for accessing the build logs is by directly looking them up in the place where they are stored by the build coordinator, the directory /var/lib/guix-build-coordinator/build-logs/. Each build corresponds to a subdirectory named after its UUID and containing the file log.gz.

Alternatively, we can imitate the behaviour of the bordeaux build farm and set up a separate nginx web server using the following service snippet:

(service nginx-service-type
  (nginx-configuration
    (server-blocks
      (list
	(nginx-server-configuration
	  (listen '("80" "[::]:80"))
	  (locations
	    (list
	      (nginx-location-configuration
		(uri "~ \"\\/build\\/([a-z0-9-]{36})/log$\"")
                (body '("alias /var/lib/guix-build-coordinator/build-logs/$1/log;"
                        "add_header Content-Type 'text/plain; charset=UTF-8';"
                        "gzip_static always;"
                        "gunzip on;")))
              (nginx-location-configuration
                (uri "/")
                (body '("proxy_pass http://localhost:8767;"
                        "proxy_http_version 1.1;"
                        "proxy_set_header Connection \"\";")))
              (nginx-location-configuration
                (uri "/events")
                (body '("proxy_pass http://localhost:8767;"
                        "proxy_http_version 1.1;"
                        "proxy_buffering off;"
                        "proxy_set_header Connection \"\";"))))))))))

The first nginx-location-configuration serves the build logs, while the other two are reverse proxies towards the pages provided by BFFE at port 8767. If the activity page is now accessed through the nginx web server at the standard port through the URL http://192.168.1.80/activity, it presents links to builds and their log files, which can be clicked on, and the uncompressed log files are shown directly in the browser.

Outlook

This was a lot of work for setting up the necessary services! But hopefully it was also a good occasion to understand how the different components interact. In the next installment we will start writing our own scripts to communicate with these services; in particular we will work with the data service to retrieve information about the packages we are interested in.

Wireguard VPN with Guix

2025-08-07T00:00:00Z

Needing a VPN

Recently I changed my ISP, and the new one uses Carrier-grade NAT, or CGNAT, by default. While this sounds fancy and professional, it is in fact even worse than conventional NAT: Not only do all my devices share the same IPv4, but I share one IPv4 with several other customers! Apparently I am only assigned a few out of the 65535 ports, and this assignment may change from day to day, which implies that I cannot connect from the outside to any of my home devices. However, I do have a separate IPv4 of my own for a virtual machine at Aquilenet, and it should be possible to use this as a trampoline to access my home through a virtual private network. We are already employing WireGuard for one of the Guix build farms, so it felt like a natural choice. Guix provides the wireguard-service-type, which is documented with all its options in the manual; but without an explanation of the general concepts behind the service it is a bit difficult to set up. The Guix Cookbook has an entry on WireGuard, but it is concerned with kernel modules and connecting to an existing WireGuard VPN, while my goal was to set one up in the first place. This turned out to be surprisingly easy.

The wireguard-tools package comes with an executable wg, and running wg --help is enough to guess how WireGuard works; essentially we need the following two subcommands:

genkey: Generates a new private key and writes it to stdout
pubkey: Reads a private key from stdin and writes a public key to stdout

Unlike other VPN, WireGuard appears to be more symmetric in the sense that it does not distinguish between servers and clients; to talk to each other, two participants just need to create a pair of public and private keys each, and then to be made aware of the other's public key. In our asymmetric situation in which only one of them has a public IPv4, we will nevertheless distinguish the server, which is reachable from everywhere thanks to its IP, and the clients hidden behind the CGNAT.

Creating key pairs

In a first step, we create a private key for the server. In Guix, secrets are (so far) not handled through the world-readable store, but as state directly on the machine, and wireguard-service-type expects by default the private key in the file /etc/wireguard/private.key. So we connect as root to the server machine and execute

mkdir /etc/wireguard
umask 077
wg genkey > /etc/wireguard/private.key

The call to umask is needed (at least with my shell settings) to placate the WireGuard warning that the private key file is world-readable, which indeed defies its purpose. The file contains a short base64 encoded number such as GEhlpFGslXfo9We9jhrXham4LztmqSmpdE4ivML4qXc=. Given the size (or rather lack thereof) of this number, it looks like WireGuard uses elliptic curve cryptography with a fixed elliptic curve and a fixed basepoint of 256 bits, so with a security level of 128 bits.

In a second step, we need to create the corresponding public key using the command

wg pubkey < /etc/wireguard/private.key

This outputs a base64 encoded number of similar size; in our example, AFL8UecS3GFX3hK8e6yWOK4s5RVrTpvTq2A0pdGuylQ=. This public key is in fact not needed by the server, but only by the clients wishing to connect to it (following a basic principle of asymmetric cryptography), so we need to stow it away in a file. But since the process of deriving the public key from a private key is deterministic, we may actually forget the public key and recreate it when needed. And notice that the public key is so short that it could even be exchanged on a postcard or over the phone.

This process of creating a key pair needs to be repeated on each client, or more generally, each participant in the VPN. Let us assume we have one client with private key 0G4uhLLeY5NYmg/FobRB0p75wMrGwmmzhuoAdfX243I= in /etc/wireguard/private.key and corresponding public key BgMzEZPUGAtbSqVPRdzgdLVhAPMLaOzHe7uNFAMVLCk=.

Setting up the server

Each participant in the WireGuard network uses a private IPv4, usually from the 10.0.0.0/8 range. We give 10.0.0.1 to the server and 10.0.0.2 to the client, and can now follow the documentation (you need to scroll down a bit) of wireguard-service-type to write the corresponding block in the Guix operating system configuration of the server:

(service wireguard-service-type
  (wireguard-configuration
    (addresses '("10.0.0.1/32"))
    (peers
      (list
        (wireguard-peer
          (name "client")
          (public-key "BgMzEZPUGAtbSqVPRdzgdLVhAPMLaOzHe7uNFAMVLCk=")
          (allowed-ips '("10.0.0.2/32")))))))

The addresses field is in fact the default; there is also an optional port field with default 51820. The peers field is a list of, well, peers in the VPN which are allowed to connect to the server (so it is in theory possible to create strange connection graphs); in this case, we register only one client peer with an arbitrary name, its public key created above, and its assigned private IP address. That is all! Now we can guix system reconfigure, and the server is ready.

Setting up the client

As stated above, in principle there does not seem to be a distinction between clients and server in WireGuard, so the operating system declaration on the client is similar to that on the server. But I think that nevertheless, it is necessary to bootstrap the network topology. And in our case, the inherent distinction between the server machine which is, say, publicly reachable on the IPv4 198.51.100.0 under the name vpn.example.org, and the client hidden by CGNAT needs to be taken into account. So we need the client to punch a hole into the NAT and to reach out to the server, which leads to the following service block:

(service wireguard-service-type
  (wireguard-configuration
    (addresses '("10.0.0.2/32"))
    (peers
      (list
        (wireguard-peer
          (name "server")
          (public-key "AFL8UecS3GFX3hK8e6yWOK4s5RVrTpvTq2A0pdGuylQ=")
          (allowed-ips '("10.0.0.1/32"))
          (endpoint "198.51.100.0:51820")
          (keep-alive 60))))))

The first fields are symmetric to the corresponding fields on the server. But the additional endpoint and keep-alive fields tell the client to connect to the server on its public IPv4 address (and the default port 51820) for the initial handshake establishing the session, and to keep it alive by reconnecting every 60 seconds. I have tried to use the host name vpn.example.org instead of the IPv4, but this ended up being resolved to an IPv6, which did not work. So guix system reconfigure the client, wait for at most one minute, and the VPN is running!

Looking behind the scenes

The following is not necessary for setting up the VPN, but it may be helpful for trouble shooting; and I was curious to see how the VPN manifested itself. Running ifconfig as root on the client, say, shows a new interface

wg0 Link encap:(hwtype unknown)
    inet addr:10.0.0.2  P-t-P:10.0.0.2  Mask:255.255.255.255
…

and running wg (again as root) shows the information about the VPN that we entered into the service description:

interface: wg0
  public key: BgMzEZPUGAtbSqVPRdzgdLVhAPMLaOzHe7uNFAMVLCk=
  private key: (hidden)
  listening port: 51820

peer: AFL8UecS3GFX3hK8e6yWOK4s5RVrTpvTq2A0pdGuylQ=
  endpoint: 198.51.100.0:51820
  allowed ips: 10.0.0.1/32
  latest handshake: 1 minute, 15 seconds ago
  transfer: 555.77 KiB received, 1.38 MiB sent
  persistent keepalive: every 1 minute

Finally, connecting from the outside!

But let us not get carried away by the beauty of technology (and cryptography), but get back to our initial concern: Connect from the outside to the machine in the home network, which we know under the name of client. This is now just a matter of two hops with ssh: First do an ssh vpn.example.org, and once on the VPN server machine, a second ssh 10.0.0.2. This can be automated by the following entry in .ssh/config on the machine from which we try to connect:

Host client
  Hostname 10.0.0.2
  ProxyJump vpn.example.org

so that from now on, ssh client will send us into the home network. Voilà, problem solved!

Epilogue

After installing my WireGuard VPN, I talked with a fellow geek from Aquilenet, who has the same ISP, and who suggested an alternative solution to me. I should call the hotline and pronounce the magic words that I would like an “IP rollback” so that servers at home become accessible. This helps passing the barrier of the first support level. The next level then initiates the “rollback”, which means going from IPv6 (plus IPv4 with CGNAT) back to regular IPv4 (without IPv6). After a few hours or days, the new more or less static (not guaranteed to be so, but actually not changing) IPv4 address is established, and IPv6 is disabled in the wireless router provided by the ISP. One can then reenable IPv6 in the router and lives in the best of all worlds – with a static IPv4 address, IPv6 and a WireGuard VPN on top of it all.

Goblins for number theory, part 4

2025-03-17T00:00:00Z

Client and server Goblins

After having introduced the basic concepts of Goblins, and in particular promises; after having looked at parallelisation over the network; and after an excursion to persistence, it is now time to get to the main topic. We would like more flexibility, as well in the behaviour of the clients as in the nature of the tasks they are handling and the control flow in the server. Clients should be able to come and go, maybe complete only one task never to be seen again (we will not handle the case of faults, however, that is, clients accepting a task and disappearing before completing it). Tasks could be heterogeneous, that is, take more or less time, or, equivalently, the clients could run on heterogeneous machines, and it would be nice to give out a new task to a client as soon as it finishes the previous one. And we would like the server to be able to work in rounds; in essence, distribute unrelated tasks corresponding to a loop, then gather the results and start with the next loop.

After David Thompson and Jessica Tallon had a look at my first solution, which had performance problems I could not explain, they came up with a much better idea, so I will present their solution without losing time in explaining what went wrong. Suffice it to say that one should avoid nesting with-vat expressions. In my experience, doing so with the same vat leads to a deadlock; doing so with different vats seems to work, but cause a severe performance penalty.

Queueing up clients, tasks and other promises

Our current solution already keeps a list of clients at the server, to which clients can register in the background. Instead of waiting until a fixed number of clients have arrived, we should be more dynamic and implement the server as follows. As long as there are tasks to be submitted and the client list is not empty, the server removes a client from the client list and submits a task to this client. If the client list is empty while there are still unsubmitted tasks, the server waits until a new client registers. So far, this scheme uses each client for exactly one task, and works if more clients register than there are tasks. The trick is to let the client do its computation, and at the end register itself again with the server as being available for the next task. My first solution used the existing ^registry actor, added an 'unregister method used by the server to retrieve an available client, and let the client call the 'register method after finishing a task. The problem with this straightforward approach is that one needs to have the server wait when no client is available, and this risks stalling everything. In a sense, we are back to the problem discussed in the first post: Goblins work with promises, and waiting for their resolution is not a Goblins concept; one should not try to master the time and write code to be executed at a specific moment, but rather define call-backs that are run when promises become true.

Jessica and David pointed out to me that since version 0.14 of Goblins, a suitable module is available in the actor library: the inbox, which is modelled after a post box that queues messages and delivers them one by one on request. Actually it rather delivers parcels, since it is a general first in, first out queue that can be filled with anything. We will replace the current clients list in a ^cell actor by an inbox that will contain client actors. The crucial difference with my home brew solution is that the level of an inbox can go below zero without blocking: If there are no elements in the queue, it nevertheless returns a promise to a future element, in our case a client actor that will register later. Thanks to promise pipelining, we can pretend that this empty promise is actually an actor and send messages to it using <-. Once a new actor registers, the promise fulfills itself, and the new actor will receive the message sent previously and act on it.

The essential modifications occur in the ^worker type actor in the client script:

(define-actor (^worker bcom server) #:self self
  (methods
    ((square x)
     (let ((res (* x x)))
          (format #t "square ~a\n" x)
          (sleep 3)
          (<- server self)
          res))
    ((finish)
     (signal-condition! end))))

After computing the result (and sleeping a little bit for testing purposes, since the tasks are so short that otherwise one client would end up grabbing all of them before we have a chance to start a second one), the client sends itself back to the server. To this purpose, it needs to know the server, which can be passed to it upon spawning; and it needs to have a notion of itself. This is why we have replaced the define by the more general define-actor, in which the optional #:self self defines a formal parameter self to later… speak to oneself! Notice that we have also modified the client so that as in the first blog posts, it does not register with its name (which thus is not passed on the command line either anymore): Using an inbox instead of the custom registry function implies that we would need to encapsulate the client actor into a composite data structure together with its name (for instance, SRFI-9 records), which is more hassle than warranted for our experimatal code. To make up for it, we let the client itself print the computing tasks it receives. The complete client script, after some shuffling around so that things are defined in the correct order, looks like this:

(use-modules (srfi srfi-1)
             (fibers conditions)
             (goblins)
             (goblins actor-lib methods)
             (goblins ocapn ids)
             (goblins ocapn captp)
             (goblins ocapn netlayer tcp-tls))

(define vat (spawn-vat))
(define net (spawn-vat))
(define end (make-condition))

(define-actor (^worker bcom server) #:self self
  (methods
    ((square x)
     (let ((res (* x x)))
          (format #t "square ~a\n" x)
          (sleep 3)
          (<- server self)
          res))
    ((finish)
     (signal-condition! end))))

(define capn
  (with-vat net (spawn-mycapn (spawn ^tcp-tls-netlayer "localhost"))))

(define server
  (with-vat vat
    (<- capn 'enliven (string->ocapn-id (second (command-line))))))

(define client
  (with-vat vat (spawn ^worker server)))
(with-vat net ($ capn 'register client 'tcp-tls))

(with-vat vat
  (<- server client))

(wait end)

In the server, we essentially replace the custom ^registry by an inbox, which results in the following code:

(use-modules (srfi srfi-1)
             (srfi srfi-26)
             (goblins)
             (goblins actor-lib cell)
             (goblins actor-lib inbox)
             (goblins actor-lib joiners)
             (goblins ocapn ids)
             (goblins ocapn captp)
             (goblins ocapn netlayer tcp-tls)
             (goblins persistence-store syrup)
             (goblins vat))

(define persistence-vat (spawn-vat))
(define persistence-registry
  (with-vat persistence-vat
    (spawn ^persistence-registry)))

(define-values (net capn)
  (spawn-persistent-vat
    (make-persistence-env #:extends (list captp-env tcp-tls-netlayer-env))
    (lambda ()
      (spawn-mycapn (spawn ^tcp-tls-netlayer "localhost")))
    (make-syrup-store "ocapn.syrup")
    #:persistence-registry persistence-registry))

(define (print-id prefix id)
  (with-vat net
    (on id
      (lambda (sref)
        (format #t "~a ~a\n"
                   prefix (ocapn-id->string sref))))))

(define-values (vat get-client put-client stop-clients)
  (spawn-persistent-vat
    (make-persistence-env
      #:extends inbox-env)
    (lambda ()
      (spawn-inbox))
    (make-syrup-store "registry.syrup")
    #:persist-on #f
    #:persistence-registry persistence-registry))

(let ((id (with-vat net ($ capn 'register put-client 'tcp-tls))))
  (print-id "Server ID" id))

(define all-clients (with-vat vat (spawn ^cell '())))
(define v '(1 2 3 4 5))
(with-vat vat
  (let
    ((clients (map (lambda (x) (<- get-client)) v)))
    ($ all-clients (append clients ($ all-clients)))
    (on (all-of* (map (cut <- <> 'square <>) clients v))
      (lambda (res)
        (format #t "~a\n" (sqrt (fold + 0 res)))
        (on (all-of* ($ all-clients))
          (lambda (c)
            (map (cut <- <> 'finish) (delete-duplicates c))))))))

(sleep 3600)

The spawn-inbox function does not return one actor, but actually three at the same time: one for adding elements into the queue, one for retrieving an element (or a promise thereof), and one for shutting the inbox down (which we will not use). The expression

(map (lambda (x) (<- get-client)) v)

creates a list of (promises to) client actors that is as long as the size of the vector. Then we send the tasks as before and use all-of* to wait for their results. There is a little subtlety for sending the 'finish messages: Since the variable clients in general does not contain a list of client actors any more, but a list of promises, we also need to use (on (all-of* …)) to retrieve the actual list of actors. We go further by memorising all clients ever encountered (with multiplicities, actually) in a separate cell all-clients. This is a bit convoluted at this point (since at the end of the script, ($ all-clients) is the same as clients), but will make things easier later. Without any extra code for sending the 'finish signal to the clients, the main part of the server script could be condensed into only a few lines:

(define v '(1 2 3 4 5))
(with-vat vat
  (on (all-of* (map (lambda (x) (<- (<- get-client) 'square x)) v))
    (lambda (res)
      (format #t "~a\n" (sqrt (fold + 0 res))))))

Notice that this solution is strictly more general than that of the previous posts: If only one client registers, it runs all the squaring tasks; if a second one arrives, it obtains every other task; and so on. And… that's it! We have parallelised a for loop which may contain tasks of differing (and a priori unknown) lengths, and it can handle the situation where clients join at any time. To handle clients that may leave after a task is completed, the framework is essentially there: Instead of having the client register again after each computing task, this could be made dependent on a condition to be checked in the client. For handling faults, that is, clients which disappear in the middle of a task, one would need to add timeouts at the server level and requeue tasks for which the result has not appeared after a reasonable waiting time, which would depend on the application. Then all-of* would not be a suitable joiner, but one could use race, which resolves as soon as one of several promises resolves. Or one could use all-of* on a list of promises created by race from a computation promise and a timeout promise, as given precisely as an example in the documentation of race. We will not pursue the topic of faults in this post, but it is clear that Goblins mechanisms could be used to solve the problem.

In any case, our current Goblins code is already more flexible than the MPI solution, which assumes that all clients are known at the beginning of the computation and do not change throughout, and which also breaks in the presence of faults.

Time for crochet: loops after loops!

A common situation is that after running one loop, one needs to start a second round that continues the computations with the intermediate results that have just been obtained. In what follows, we will modify the server script accordingly, while keeping the client script unmodified, which may be seen as a sign that the architecture developed so far makes sense.

For instance, the following sequential code computes the L4-norm of a vector, that is, the fourth root of the sum of the fourth powers of its entries:

(use-modules (srfi srfi-1))
(define (square x) (* x x))
(define v '(1 2 3 4 5))
(define w (map square v))
(define res (map square w))
(format #t "~a\n" (sqrt (sqrt (fold + 0 res))))

It consists of two loops, one for squaring each entry in v and putting the results into w, and a second one for squaring the entries in w (which effectively computes the fourth powers of the entries in v).

This can be goblinified quite naturally by nesting the task submission and (on (all-of* …)) handling of the results. Without 'finish signals, this results in the following code:

(define v '(1 2 3 4 5))
(with-vat vat
  (on (all-of* (map (lambda (x) (<- (<- get-client) 'square x)) v))
    (lambda (w)
      (on (all-of* (map (lambda (x) (<- (<- get-client) 'square x)) w))
        (lambda (res)
          (format #t "~a\n" (sqrt (sqrt (fold + 0 res)))))))))

Including 'finish handling, the following server script is not the shortest solution, but its symmetries will be helpful in the next section. To make the code more readable, we have moved some (repetitive) code into the functions submit-square-jobs and submit-finish-jobs.

(use-modules (srfi srfi-1)
             (srfi srfi-26)
             (goblins)
             (goblins actor-lib cell)
             (goblins actor-lib inbox)
             (goblins actor-lib joiners)
             (goblins ocapn ids)
             (goblins ocapn captp)
             (goblins ocapn netlayer tcp-tls)
             (goblins persistence-store syrup)
             (goblins vat))

(define persistence-vat (spawn-vat))
(define persistence-registry
  (with-vat persistence-vat
    (spawn ^persistence-registry)))

(define-values (net capn)
  (spawn-persistent-vat
    (make-persistence-env #:extends (list captp-env tcp-tls-netlayer-env))
    (lambda ()
      (spawn-mycapn (spawn ^tcp-tls-netlayer "localhost")))
    (make-syrup-store "ocapn.syrup")
    #:persistence-registry persistence-registry))

(define (print-id prefix id)
  (with-vat net
    (on id
      (lambda (sref)
        (format #t "~a ~a\n"
                   prefix (ocapn-id->string sref))))))

(define-values (vat get-client put-client stop-clients)
  (spawn-persistent-vat
    (make-persistence-env
      #:extends inbox-env)
    (lambda ()
      (spawn-inbox))
    (make-syrup-store "registry.syrup")
    #:persist-on #f
    #:persistence-registry persistence-registry))

(let ((id (with-vat net ($ capn 'register put-client 'tcp-tls))))
  (print-id "Server ID" id))

(define all-clients (with-vat vat (spawn ^cell '())))

(define (submit-square-jobs v)
  (let ((clients (map (lambda (x) (<- get-client)) v)))
    ($ all-clients (append clients ($ all-clients)))
    (map (cut <- <> 'square <>) clients v)))

(define (submit-finish-jobs clients)
  (map (cut <- <> 'finish) (delete-duplicates clients)))

(define v '(1 2 3 4 5))
(with-vat vat
  (on (all-of* (submit-square-jobs  v))
    (lambda (w)
      (on (all-of* (submit-square-jobs w))
        (lambda (res)
          (format #t "~a\n" (sqrt (sqrt (fold + 0 res))))
          (on (all-of* ($ all-clients))
            submit-finish-jobs))))))

(sleep 3600)

Untangling the threads: macros to the rescue

The last block of the server code now clearly shows a recurring pattern:

(on (all-of* SUBMIT SOME JOBS)
  (lambda (VAR)
    DO SOMETHING WITH THE RESULT IN VAR

which is actually nested, since handling the results of the first round requires to run the same pattern for the second round of job submissions. Now a pattern can be handled by a Guile macro, for instance as follows:

(define-syntax submit-reduce
  (syntax-rules ()
    ((submit-reduce submit v reduce ...)
     (on (all-of* submit)
       (lambda (v)
         (begin reduce ...))))))

The line following the syntax-rules () contains a pattern to be matched; the remainder of the macro is the Guile code above, with placeholders replaced by parts of the matched pattern. The first argument of the macro is a single expression corresponding to SUBMIT SOME JOBS; if several expressions are needed, they can be transformed into only one using let*, for instance. The second argument is the (formal) variable name VAR. All remaining arguments (of which there may be zero) correspond to DO SOMETHING WITH THE RESULT IN VAR; these will in general use the formal variable.

Using this macro, the main block of the server script can be compressed as follows:

(define v '(1 2 3 4 5))
(with-vat vat
  (submit-reduce (submit-square-jobs  v) w
    (submit-reduce (submit-square-jobs w) res
      (format #t "~a\n" (sqrt (sqrt (fold + 0 res))))
      (on (all-of* ($ all-clients))
        submit-finish-jobs))))

It is also possible to let the macro itself handle the nesting as follows:

(define-syntax submit-reduce
  (syntax-rules ()
    ((submit-reduce reduce)
     reduce)
    ((submit-reduce submit v reduce ...)
     (on (all-of* submit)
       (lambda (v)
         (submit-reduce reduce ...))))))

If the macro is called with at least three arguments, then the second pattern (submit-reduce submit v reduce ...) is matched. The first argument (a single expression) is considered to be the job submission phase, the second argument the variable name for the results of the first jobs; then the macro is called recursively, and more job submission phases, alternated with variable names, are expected; in the end, when only one argument remains, the first pattern (submit-reduce reduce) is matched, which corresponds to the handling of the results of the final round of job submissions. So to work, the macro requires an odd number of arguments, otherwise it raises an error (using just one argument is possible, but makes no sense). With this macro, the main server block looks as follows:

(define v '(1 2 3 4 5))
(with-vat vat
  (submit-reduce
    (submit-square-jobs v) w
    (submit-square-jobs w) res
    (begin
      (format #t "~a\n" (sqrt (sqrt (fold + 0 res))))
      (on (all-of* ($ all-clients))
        submit-finish-jobs))))

Notice that we needed to wrap the final reduction into (begin …) since it consists of several expressions. This macro, without its additional nesting, makes the sequence of submitting a series of tasks, submitting a new series of tasks depending on the results of the previous series, and so on, until the final result is handled through a side effect (to break out of the promises), quite clear. When exactly three arguments are given, both macros are equivalent, so that it is still possible to manually nest the macro invocations, and the second macro is more powerful than the first one (except that the first one admits several Guile expressions for the reduction phase).

To illustrate the simplicity with which the pattern continues, here is the complete server script for computing the L8-norm of a vector, that is, the eightth root of the sum of the eigtth powers of its entries:

(use-modules (srfi srfi-1)
             (srfi srfi-26)
             (goblins)
             (goblins actor-lib cell)
             (goblins actor-lib inbox)
             (goblins actor-lib joiners)
             (goblins ocapn ids)
             (goblins ocapn captp)
             (goblins ocapn netlayer tcp-tls)
             (goblins persistence-store syrup)
             (goblins vat))

(define persistence-vat (spawn-vat))
(define persistence-registry
  (with-vat persistence-vat
    (spawn ^persistence-registry)))

(define-values (net capn)
  (spawn-persistent-vat
    (make-persistence-env #:extends (list captp-env tcp-tls-netlayer-env))
    (lambda ()
      (spawn-mycapn (spawn ^tcp-tls-netlayer "localhost")))
    (make-syrup-store "ocapn.syrup")
    #:persistence-registry persistence-registry))

(define (print-id prefix id)
  (with-vat net
    (on id
      (lambda (sref)
        (format #t "~a ~a\n"
                   prefix (ocapn-id->string sref))))))

(define-values (vat get-client put-client stop-clients)
  (spawn-persistent-vat
    (make-persistence-env
      #:extends inbox-env)
    (lambda ()
      (spawn-inbox))
    (make-syrup-store "registry.syrup")
    #:persist-on #f
    #:persistence-registry persistence-registry))

(let ((id (with-vat net ($ capn 'register put-client 'tcp-tls))))
  (print-id "Server ID" id))

(define all-clients (with-vat vat (spawn ^cell '())))

(define (submit-square-jobs v)
  (let ((clients (map (lambda (x) (<- get-client)) v)))
    ($ all-clients (append clients ($ all-clients)))
    (map (cut <- <> 'square <>) clients v)))

(define (submit-finish-jobs clients)
  (map (cut <- <> 'finish) (delete-duplicates clients)))

(define-syntax submit-reduce
  (syntax-rules ()
    ((submit-reduce reduce)
     reduce)
    ((submit-reduce submit v reduce ...)
     (on (all-of* submit)
       (lambda (v)
         (submit-reduce reduce ...))))))

(define v '(1 2 3 4 5))
(with-vat vat
  (submit-reduce
    (submit-square-jobs v) w
    (submit-square-jobs w) t
    (submit-square-jobs t) res
    (begin
      (format #t "~a\n" (sqrt (sqrt (sqrt (fold + 0 res)))))
      (on (all-of* ($ all-clients))
        submit-finish-jobs))))

(sleep 3600)

(Preliminary) conclusion

At this point, the goal set out at the beginning of this series of blog posts is met. We have developed a client and server structure in which the clients register with the server and the server hands them computation tasks that correspond to a sequence of loops. As already said above, the result is even a bit more flexible than with MPI: The number of clients need not be known and communicated to the server in advance, but clients can come and go, as long as they do not vanish in the middle of a task. And Goblins make it possible to do so over the Internet, either with TCP/TLS or even through the Tor network.

Goblins for number theory, part 3

2025-03-07T00:00:00Z

Ending and persisting

In previous posts we have seen how to solve our toy problem of computing the euclidian length of a vector in a distributed fashion using Goblins, with a client script that runs in several copies, carries out most of the work and reports back to a server script, which collects the partial results into a solution to the problem. The clients could in principle live on distant machines and communicate over the Tor network. For testing in a local setting, however, letting them run on the same machine as the server and communicating over TCP turns out to be more efficient. So far, our architecture is rather inflexible: We assume that the server knows the number of participating clients beforehand, and that all tasks take more or less the same time so that distributing them evenly to the clients is an optimal scheduling strategy. The logical next step is to overcome these limitations. My initial solution for a more general framework, however, turned out to be very inefficient. Jessica Tallon and David Thompson of the Spritely Institute (many thanks to them!) kindly had a look at it and came up with a much better solution; but our discussions also helped me understand Goblins better and inspired ideas on how to improve the current client and server scripts. So before going for more generality in the next post, let us do a pirouette with the current framework and also explore some interesting side tracks that did not make it into the previous post.

Spring cleaning

Before doing anything substantial, let us clean up a few things in the current code. The main actor in the server script is currently defined through the type ^register as follows:

(define clients (with-vat vat (spawn ^cell '())))
(define (^register bcom)
  (lambda (id)
    ($ clients (cons (<- mycapn 'enliven id)
                     ($ clients)))
    (print-id "Registered" id)))
(define register (with-vat vat (spawn ^register)))

It captures the clients variable in the closure defined by lambda, which works, but requires the variables to be defined in this order. A more elegant solution is to pass clients as an argument. At the same time, we take the opportunity to rename the verb register to the noun registry.

(define (^registry bcom clients)
  (lambda (id)
    ($ clients (cons (<- mycapn 'enliven id)
                     ($ clients)))
    (print-id "Registered" id)))
(define clients (with-vat vat (spawn ^cell '())))
(define registry (with-vat vat (spawn ^registry clients)))

Let us also get rid of some “overgoblinification”; indeed the actor of type ^len in the server can be replaced by a simple function, or (since the Goblins promises force us to work with side effects anyway) by sequential code. We end up with the following server script server.scm:

(use-modules (srfi srfi-1)
             (goblins)
             (goblins actor-lib cell)
             (goblins actor-lib joiners)
             (goblins actor-lib methods)
             (goblins ocapn ids)
             (goblins ocapn captp)
             (goblins ocapn netlayer tcp-tls))

(define vat (spawn-vat))
(define net (spawn-vat))

(define (print-id prefix id)
  (with-vat net
    (on id
      (lambda (sref)
        (format #t "~a ~a\n"
                   prefix (ocapn-id->string sref))))))

(define capn
   (with-vat net (spawn-mycapn (spawn ^tcp-tls-netlayer "localhost"))))

(define (^registry bcom clients)
  (lambda (id)
    ($ clients (cons (<- capn 'enliven id)
                     ($ clients)))
    (print-id "Registered" id)))

(define clients (with-vat vat (spawn ^cell '())))
(define registry (with-vat vat (spawn ^registry clients)))
(let ((id (with-vat net ($ capn 'register registry 'tcp-tls))))
  (print-id "Server ID" id))

(while (not (eq? (length (with-vat vat ($ clients))) 2))
       (sleep 1))

(define v '(1 2 3 4 5))
(with-vat vat
  (while (< (length ($ clients)) (length v))
     (let ((c ($ clients)))
       ($ clients (append c c)))))
(with-vat vat
  (on (all-of* (map <- ($ clients) v))
      (lambda (res)
        (format #t "~a\n" (sqrt (fold + 0 res))))))

(sleep 3600)

and the following client script client.scm:

(use-modules (srfi srfi-1)
             (goblins)
             (goblins ocapn ids)
             (goblins ocapn captp)
             (goblins ocapn netlayer tcp-tls))

(define vat (spawn-vat))
(define net (spawn-vat))

(define (^square bcom)
  (lambda (x)
    (* x x)))
(define client
  (with-vat vat (spawn ^square)))

(define (print-id prefix id)
  (with-vat net
    (on id
      (lambda (sref)
        (format #t "~a ~a\n"
                   prefix (ocapn-id->string sref))))))

(define capn
  (with-vat net (spawn-mycapn (spawn ^tcp-tls-netlayer "localhost"))))
(define id
  (with-vat net ($ capn 'register client 'tcp-tls)))
(print-id "Client ID" id)

(define server
  (with-vat vat
    (<- capn 'enliven (string->ocapn-id (second (command-line))))))

(with-vat vat
  (on id
    (lambda (id)
      (<- server id))))

(sleep 3600)

Now run again

guile server.scm

in one terminal and two copies of the client script as

guile client.scm 'ocapn://…'

in two other terminals, where the ocapn URI has been replaced by the one printed by the server, to compute the same result as before.

Passing actors around

After going through the CapTP tutorial, I was under the impression that the only way to create a handle on an actor on a different machine was by obtaining its sturdyref ID and “enlivening” this ID locally. Currently the server script prints its ID, which the client script obtains as an argument when invoked from the command line. This enables the client to enliven the server and to send its ID to the server when registering by a <- call; then the server enlivens the client. It turns out, however, that it is also possible to directly send actors instead of their IDs through <-. Printing and copy-pasting IDs is still necessary for bootstrapping, but once a spanning tree is generated in this manner between all participating scripts, it is possible to obtain a complete communication graph by just sending actors along these bootstrapped network edges.

We would still like the client to somehow present itself to the server with a name, so that the server can print who connects to it and thus make debugging easier. If we drop the ocapn ID, then the client can use a pet name, a string that we pass as an additional argument on the command line. The server needs only minimal modifications:

(use-modules (srfi srfi-1)
             (goblins)
             (goblins actor-lib cell)
             (goblins actor-lib joiners)
             (goblins actor-lib methods)
             (goblins ocapn ids)
             (goblins ocapn captp)
             (goblins ocapn netlayer tcp-tls))

(define vat (spawn-vat))
(define net (spawn-vat))

(define (print-id prefix id)
  (with-vat net
    (on id
      (lambda (sref)
        (format #t "~a ~a\n"
                   prefix (ocapn-id->string sref))))))

(define capn
   (with-vat net (spawn-mycapn (spawn ^tcp-tls-netlayer "localhost"))))

(define (^registry bcom clients)
  (lambda (client name)
    ($ clients (cons client ($ clients)))
    (format #t "Registered ~a\n" name)))

(define clients (with-vat vat (spawn ^cell '())))
(define registry (with-vat vat (spawn ^registry clients)))
(let ((id (with-vat net ($ capn 'register registry 'tcp-tls))))
  (print-id "Server ID" id))

(while (not (eq? (length (with-vat vat ($ clients))) 2))
       (sleep 1))

(define v '(1 2 3 4 5))
(with-vat vat
  (while (< (length ($ clients)) (length v))
     (let ((c ($ clients)))
       ($ clients (append c c)))))
(with-vat vat
  (on (all-of* (map <- ($ clients) v))
      (lambda (res)
        (format #t "~a\n" (sqrt (fold + 0 res))))))

(sleep 3600)

Notice the additional argument name for the ^registry actor, which is used for announcing arriving clients instead of their ocapn ID. (In this implementation we forget the name of a client immediately; it would make sense to somehow keep it, either by remembering it directly in ^square or by having the server memorise it in its client list.) Instead of enlivening an ID and adding the resulting actor to the clients list, the server adds the client actor directly. The client modifications are also straightforward and simplify the script considerably:

(use-modules (srfi srfi-1)
             (goblins)
             (goblins ocapn ids)
             (goblins ocapn captp)
             (goblins ocapn netlayer tcp-tls))

(define vat (spawn-vat))
(define net (spawn-vat))

(define (^square bcom)
  (lambda (x)
    (* x x)))
(define client
  (with-vat vat (spawn ^square)))

(define capn
  (with-vat net (spawn-mycapn (spawn ^tcp-tls-netlayer "localhost"))))
(with-vat net ($ capn 'register client 'tcp-tls))

(define name (second (command-line)))

(define server
  (with-vat vat
    (<- capn 'enliven (string->ocapn-id (third (command-line))))))

(with-vat vat
  (<- server client name))

(sleep 3600)

Now start the server as usual, and two clients as

guile client.scm Alice 'ocapn://…'
guile client.scm Bob 'ocapn://…'

to see the familiar result.

Being methodical

As it will be useful later on, let us replace the workhorse in the client, the ^square actor with only one possible action (squaring a number that is sent to it) by an implementation with potentially more actions. To do so, we use methods from Goblin actor libs, which dispatch actions using an additional symbol. So

(define (^square bcom)
  (lambda (x)
    (* x x)))
(define client
  (with-vat vat (spawn ^square)))

becomes

(use-module (goblins actor-lib methods)
…
(define (^worker bcom)
  (methods
    ((square x)
     (* x x))))
(define client
  (with-vat vat (spawn ^worker)))

Inside the server, we now need to change calls of the form

(<- client x)

by adding an additional symbol to

(<- client 'square x)

This is made more complicated since they appear inside map:

(map <- ($ clients) v)

The solution is to change the <- function, which now takes three arguments (a client, a symbol and a number) into a function with only two arguments by fixing the middle argument to 'square. This can be done using SRFI-26 cut; it takes the function name and for each argument of the function either a fixed value, or the placeholder <> indicating that this argument should be kept as such. In our case, this gives

(map (cut <- <> 'square <>) ($ clients) v))

So altogether, here is our current server:

(use-modules (srfi srfi-1)
             (srfi srfi-26)
             (goblins)
             (goblins actor-lib cell)
             (goblins actor-lib joiners)
             (goblins actor-lib methods)
             (goblins ocapn ids)
             (goblins ocapn captp)
             (goblins ocapn netlayer tcp-tls))

(define vat (spawn-vat))
(define net (spawn-vat))

(define (print-id prefix id)
  (with-vat net
    (on id
      (lambda (sref)
        (format #t "~a ~a\n"
                   prefix (ocapn-id->string sref))))))

(define capn
   (with-vat net (spawn-mycapn (spawn ^tcp-tls-netlayer "localhost"))))

(define (^registry bcom clients)
  (lambda (client name)
    ($ clients (cons client ($ clients)))
    (format #t "Registered ~a\n" name)))

(define clients (with-vat vat (spawn ^cell '())))
(define registry (with-vat vat (spawn ^registry clients)))
(let ((id (with-vat net ($ capn 'register registry 'tcp-tls))))
  (print-id "Server ID" id))

(while (not (eq? (length (with-vat vat ($ clients))) 2))
       (sleep 1))

(define v '(1 2 3 4 5))
(with-vat vat
  (while (< (length ($ clients)) (length v))
     (let ((c ($ clients)))
       ($ clients (append c c)))))
(with-vat vat
  (on (all-of* (map (cut <- <> 'square <>) ($ clients) v))
      (lambda (res)
        (format #t "~a\n" (sqrt (fold + 0 res))))))

(sleep 3600)

and here our current client:

(use-modules (srfi srfi-1)
             (goblins)
             (goblins actor-lib methods)
             (goblins ocapn ids)
             (goblins ocapn captp)
             (goblins ocapn netlayer tcp-tls))

(define vat (spawn-vat))
(define net (spawn-vat))

(define (^worker bcom)
  (methods
    ((square x)
     (* x x))))
(define client
  (with-vat vat (spawn ^worker)))

(define capn
  (with-vat net (spawn-mycapn (spawn ^tcp-tls-netlayer "localhost"))))
(with-vat net ($ capn 'register client 'tcp-tls))

(define name (second (command-line)))

(define server
  (with-vat vat
    (<- capn 'enliven (string->ocapn-id (third (command-line))))))

(with-vat vat
  (<- server client name))

(sleep 3600)

Everything has an end, but Goblins

It is mildly annoying that the scripts run forever (well, for one hour…) and need to be stopped with <ctrl-c>. But it is somewhat difficult to decide when to stop: In both our scripts, the control flow reaches the end of the programs, while Goblins are still working in the background through promises. It is possible to use conditions from Guile Fibers, as inspired by the chat example in the Goblins documentation. Since Fibers are a basic ingredient of Goblins in Guile, they do not need to be installed separately. We can modify the client as follows:

(use-module (fibers conditions)
…
(define end (make-condition))
…
(define (^worker bcom)
  (methods
    ((square x)
     (* x x))
    ((finish)
     (signal-condition! end))))
…
(wait end)

First we import the (fibers conditions) module. Then we create the “condition” end. We use signal-condition! to signal, well, that the condition has been fulfilled. And we replace sleeping by waiting for the condition. The signalling is encapsulated in a new method 'finish of the ^worker actor, which can be called from the server as

(map (cut <- <> 'finish) ($ clients))

after the result of the computations has been printed. This results in the following client script:

(use-modules (srfi srfi-1)
             (fibers conditions)
             (goblins)
             (goblins actor-lib methods)
             (goblins ocapn ids)
             (goblins ocapn captp)
             (goblins ocapn netlayer tcp-tls))

(define vat (spawn-vat))
(define net (spawn-vat))
(define end (make-condition))

(define (^worker bcom)
  (methods
    ((square x)
     (* x x))
    ((finish)
     (signal-condition! end))))
(define client
  (with-vat vat (spawn ^worker)))

(define capn
  (with-vat net (spawn-mycapn (spawn ^tcp-tls-netlayer "localhost"))))
(with-vat net ($ capn 'register client 'tcp-tls))

(define name (second (command-line)))

(define server
  (with-vat vat
    (<- capn 'enliven (string->ocapn-id (third (command-line))))))

(with-vat vat
  (<- server client name))

(wait end)

With the server script modified suitably as explained above, the clients now end correctly, but the server crashes after printing the result of the computations. A hasty decision we took earlier comes back to haunt us now: Since there are more tasks than clients, we have filled the clients list with duplicates of the client actors so as to send multiple 'square messages to the same actor; but now we send multiple 'finish messages to clients that have stopped running after the first such message, resulting in a scary error on the server side that boils down to &non-continuable. To reach this correct conclusion more gracefully, we take another hasty decision and deduplicate the clients list when calling finish:

(map (cut <- <> 'finish) (delete-duplicates ($ clients)))

An an excuse for our laziness in not looking for a more elegant solution, we remark that anyway this part will be reworked later to obtain a more flexible client queue.

I have not found a similar approach to also have the server end gracefully. If one places signal-condition! in the code right after sending the 'finish messages to the clients, then the clients do not end, since it turns out that the server finishes so fast that the messages are not actually sent. If one tries to wait for the promise coming out of the 'finish calls, then this also fails, since the finished clients cannot send back a function value any more. So I keep the sleep in the end and make it just a bit shorter. The current server.scm then looks like this:

(use-modules (srfi srfi-1)
             (srfi srfi-26)
             (goblins)
             (goblins actor-lib cell)
             (goblins actor-lib joiners)
             (goblins actor-lib methods)
             (goblins ocapn ids)
             (goblins ocapn captp)
             (goblins ocapn netlayer tcp-tls))

(define vat (spawn-vat))
(define net (spawn-vat))

(define (print-id prefix id)
  (with-vat net
    (on id
      (lambda (sref)
        (format #t "~a ~a\n"
                   prefix (ocapn-id->string sref))))))

(define capn
   (with-vat net (spawn-mycapn (spawn ^tcp-tls-netlayer "localhost"))))

(define (^registry bcom clients)
  (lambda (client name)
    ($ clients (cons client ($ clients)))
    (format #t "Registered ~a\n" name)))

(define clients (with-vat vat (spawn ^cell '())))
(define registry (with-vat vat (spawn ^registry clients)))
(let ((id (with-vat net ($ capn 'register registry 'tcp-tls))))
  (print-id "Server ID" id))

(while (not (eq? (length (with-vat vat ($ clients))) 2))
       (sleep 1))

(define v '(1 2 3 4 5))
(with-vat vat
  (while (< (length ($ clients)) (length v))
     (let ((c ($ clients)))
       ($ clients (append c c)))))
(with-vat vat
  (on (all-of* (map (cut <- <> 'square <>) ($ clients) v))
      (lambda (res)
        (format #t "~a\n" (sqrt (fold + 0 res)))
        (map (cut <- <> 'finish) (delete-duplicates ($ clients))))))

(sleep 10)

Résistez ! euh, persistez !

Another annoyance in the current code is that the ocapn ID of the server changes every time it is started, so that there is a lot of copy-pasting for starting the clients. This turns from a minor annoyance into a problem when different clients are supposed to be started independently all over the Internet, and the ocapn ID is the de facto credential to enable connections. Then a restart of the server script for any reason, be it a power outage or an update, requires to communicate the new ID to all participants. From the name of it, it sounds as if persistence could come to the rescue. We only need to persist the server. In a first step, we add a bit of boilerplate, taken from the documentation of persistent vats; this seems to be required when several vats with cross-references to each other are to be persisted, but cannot do any harm in general.

(use-module (goblins vat)
…
(define persistence-vat (spawn-vat))
(define persistence-registry
  (with-vat persistence-vat
    (spawn ^persistence-registry)))

Then we follow the example on persistence in the documentation of the TCP netlayer (after correcting a small error in the documentation for version 0.15, which has been updated in the meantime) and replace

(define net (spawn-vat))
(define capn
   (with-vat net (spawn-mycapn (spawn ^tcp-tls-netlayer "localhost"))))

(use-module (goblins persistence-store syrup)
…
(define-values (net capn)
  (spawn-persistent-vat
    (make-persistence-env #:extends (list captp-env tcp-tls-netlayer-env))
    (lambda ()
      (spawn-mycapn (spawn ^tcp-tls-netlayer "localhost")))
    (make-syrup-store "ocapn.syrup")
    #:persistence-registry persistence-registry))

The spawn-persistent-vat returns a number of values; the first one is a new vat, the other ones are created by the lambda expression and correspond to actors in the vat which are to be persisted (more precisely, they form the roots of the corresponding graph). A persistence environment is passed as the first argument; it “knows” how to store the different types of actors. In this case, we store to a file named ocapn.syrup, where syrup is the Goblins internal file format.

It is instructive to run the server and to inspect the ocapn ID it prints. The general format seems to be ocapn://….tcp-tls/s/…?host=localhost&port=… where the first ellipsis consists of 52 lower case letters and digits (a 256 bit hash encoded in base 32?), the second ellipsis consists of 43 lower and upper case letters, digits and symbols (a 256 bit hash encoded in base 64?), and the third ellipsis is a random port. Previously, all three would change when invoking the script. Now the sequence in the place of the first ellipsis as well as the port remain fixed.

So we need to persist more, in particular the actor that is registered in the network layer. So we replace

(define (^registry bcom clients) …)
(define vat (spawn-vat))
(define clients (with-vat vat (spawn ^cell '())))
(define registry (with-vat vat (spawn ^registry clients)))

(define-actor (^registry bcom clients) …)
(define-values (vat clients registry)
  (spawn-persistent-vat
    (make-persistence-env
      (list (list '((registry) ^registry) ^registry))
      #:extends cell-env)
    (lambda ()
      (let ((clients (spawn ^cell '())))
        (values
          clients
          (spawn ^registry clients))))
    (make-syrup-store "registry.syrup")
    #:persistence-registry persistence-registry))

Notice the use of define-actor instead of define, which appears to be necessary to achieve persistence. Besides the cell actor known to Goblins from the actor-lib, we also need to declare our self-defined actor of type ^registry in the persistence environment; this is obtained by the rather indigest boiler plate line creating nested lists. We use a second file, registry.syrup, to store this actor.

However, this fails miserably, as the server crashes with an error message containing keywords such as vat-churn and vat-maybe-persist-changed-objs!. What happens exactly seems to depend on timing. In this case there is a 176 byte file registry.syrup containing a few strings and binary data. I suppose it stores the empty client list and the corresponding registry. After clients register, there is a “churn” (which I understand as the vat taking a break after a turn is over), and the persistence system tries to update the file. However, the client list now contains an actor coming from the client script, that is, coming over the network from potentially a different machine. Since this is not under the control of the local script, it cannot be stored.

There is apparently a very simple workaround. The spawn-persistent-vat function admits on optional parameter #:persist-on; if this is changed from the default 'churn to something else, then the vat changes are not stored at each churn. In effect, the vat is only stored once in the beginning, and keeps an empty client list forever. This is actually exactly what we need, an empty client list at each restart of the server. So we end up with the following server.scm:

(use-modules (srfi srfi-1)
             (srfi srfi-26)
             (goblins)
             (goblins actor-lib cell)
             (goblins actor-lib joiners)
             (goblins actor-lib methods)
             (goblins ocapn ids)
             (goblins ocapn captp)
             (goblins ocapn netlayer tcp-tls)
             (goblins persistence-store syrup)
             (goblins vat))

(define persistence-vat (spawn-vat))
(define persistence-registry
  (with-vat persistence-vat
    (spawn ^persistence-registry)))

(define-values (net capn)
  (spawn-persistent-vat
    (make-persistence-env #:extends (list captp-env tcp-tls-netlayer-env))
    (lambda ()
      (spawn-mycapn (spawn ^tcp-tls-netlayer "localhost")))
    (make-syrup-store "ocapn.syrup")
    #:persistence-registry persistence-registry))

(define (print-id prefix id)
  (with-vat net
    (on id
      (lambda (sref)
        (format #t "~a ~a\n"
                   prefix (ocapn-id->string sref))))))

(define-actor (^registry bcom clients)
  (lambda (client name)
    ($ clients (cons client ($ clients)))
    (format #t "Registered ~a\n" name)))

(define-values (vat clients registry)
  (spawn-persistent-vat
    (make-persistence-env
      (list (list '((registry) ^registry) ^registry))
      #:extends cell-env)
    (lambda ()
      (let ((clients (spawn ^cell '())))
        (values
          clients
          (spawn ^registry clients))))
    (make-syrup-store "registry.syrup")
    #:persist-on #f
    #:persistence-registry persistence-registry))

(let ((id (with-vat net ($ capn 'register registry 'tcp-tls))))
  (print-id "Server ID" id))

(while (not (eq? (length (with-vat vat ($ clients))) 2))
       (sleep 1))

(define v '(1 2 3 4 5))
(with-vat vat
  (while (< (length ($ clients)) (length v))
     (let ((c ($ clients)))
       ($ clients (append c c)))))
(with-vat vat
  (on (all-of* (map (cut <- <> 'square <>) ($ clients) v))
      (lambda (res)
        (format #t "~a\n" (sqrt (fold + 0 res)))
        (map (cut <- <> 'finish) (delete-duplicates ($ clients))))))

(sleep 10)

It may be prudent now to remove all .syrup files from previous failed attempts. Running a server and two client scripts computes the desired result as before. But now one notices that upon restarting the server script, it prints the exact same ocapn ID as before. So the clients can also be restarted with the exact same commands, and no more copy-pasting is needed.

Goblins for number theory, part 2

2025-02-25T00:00:00Z

Parallel Goblins

After seeing how to use the programming concepts of Goblins for a toy problem the structure of which resembles algorithms encountered in number theory, let us turn our attention to parallelising, or rather distributing the code. We keep the running example of computing the length of a vector, by giving out the tasks of squaring to the clients, and leaving the task of adding up the squares and taking the final square root to the server.

Networking

Communication in Goblins is abstracted over what is called the “Object Capabilities Network”, or “OCapN”. This somewhat frightening term simply means that a function in one script may call functions in another script running elsewhere in the network.

Goblins suggests to use Tor as the underlying network. Indeed after setting up a Tor daemon as described in the Goblins documentation on my laptop, the provided example of a chat client Alice talking to a chat server Bob works directly out of the box. This should also make it relatively easy to run distributed projects over the Internet, which would fit the idea of using Goblins for popular science projects.

On the other hand, institutional computing clusters tend to limit network access, sometimes even blocking outgoing HTTP requests to servers outside a whitelist. So it is unlikely that the Tor approach will work in this setting. Also it appears that Tor needs to have access to the Internet for bootstrapping: The chat script does not run purely locally after turning off Internet access. It may be possible to set up Tor in a specific way to cover such local use cases, but so far my knowledge of Tor is limited to what is described in the Goblins documentation. The documentation points to the possibility of using TCP. This requires that the participating nodes know each other's IP address or hostname, which sounds restrictive, but since I am currently using MPI over TCP, OpenMPI seems to somehow be able to determine these addresses, so it should also be a feasible option with Goblins. But for the time being let us assume that we are working with a machine that has access to the Tor network after setting up the Tor daemon as taught by the Goblins documentation; we will come back to the TCP setting below.

From chatting to computing

When saying that OCapN enables a function to call functions running somewhere else in the Tor network, one should more precisely use the term “actor” instead of “function”; and as seen before, these do not return values, but promises that resolve to the desired values. But it is conceptually helpful to think of calls to outsourced functions. So in our very simple model inspired by algorithmic number theory, we will have a client script that runs in a number of identical copies, and a server script that calls functions defined in the clients. This is in fact much easier to programme than with MPI, where the exchange of function arguments and results requires explicit MPISend and matching MPIReceive statements in the server and the client, and where furthermore complex data types need to be serialised by hand since the communication functions work only with basic, scalar types. Finally it is necessary to carefully and explicitly craft the control flows of the different programs exchanging data so that indeed the data sending statements exactly match the data receiving statements; otherwise there will be a deadlock. In the Goblins framework, this is all implicit. As an end result a distributed code does not look very different from the corresponding serial code.

But we still need to make a few things explicit: First of all, the different running scripts need to connect to the network. And the functions to be called remotely need to obtain a unique identifier and advertise it, and the caller needs to know this identifier to make the call. Luckily in our setting, most of the corresponding code can be considered as copy-pastable boilerplate.

Indeed the chat example can be transposed to our running example of vector lengths almost immediately.

Let us start with the client, to be put into a file client.scm (compared with the chat example, the client and server roles are reversed):

(use-modules (srfi srfi-1)
             (goblins)
             (goblins ocapn ids)
             (goblins ocapn captp)
             (goblins ocapn netlayer onion))

(define vat (spawn-vat))
(define net (spawn-vat))

;; Define the client functionality.
(define (^square bcom)
  (lambda (x)
    (* x x)))
(define client
  (with-vat vat (spawn ^square)))

;; Helper function for printing IDs.
(define (print-id prefix id)
  (with-vat net
    (on id
      (lambda (sref)
        (format #t "~a ~a\n"
                   prefix (ocapn-id->string sref))))))

;; Create a communicator.
(define mycapn
  (with-vat net (spawn-mycapn (spawn ^onion-netlayer))))
;; Create an ID for the client and print it.
(define id
  (with-vat net ($ mycapn 'register client 'onion)))
(print-id "Client ID" id)

;; Wait for requests.
(sleep 3600)

The chat example uses two vats to separate the networking part and the actual functionality. This does not seem to be strictly necessary (the examples also work when everything is put into the same vat); but if I understand correctly, each vat corresponds to a separate, concurrent event loop, so having several vats might help to prevent deadlocks and possibly speed things up by separating communication and computation, so I follow the example and declare two vats from the start, net for everything network related and vat for everything else. The client actor in the main vat is defined as before through the function computing a square.

A network connection mycapn is defined, the client actor is registered with it and its network ID id is obtained through some magic incantations. Before version 0.15.0 of Goblins, id used to be a value, but now it is a promise. So before printing it as a string by applying the ocapn-id->string function, one needs to wait for the resolution of the promise; this is moved into the helper function print-id. This string value will be used to communicate the ID manually to the server later on.

Finally, we just wait for requests to compute squares (the chat example has a more sophisticated approach to waiting using Guile fibers, but sleep is enough for illustration purposes).

The corresponding server follows, to be put into a file server.scm:

(use-modules (srfi srfi-1)
             (goblins)
             (goblins actor-lib joiners)
             (goblins actor-lib let-on)
             (goblins ocapn ids)
             (goblins ocapn captp)
             (goblins ocapn netlayer onion))

(define vat (spawn-vat))
(define net (spawn-vat))

(define mycapn
   (with-vat net (spawn-mycapn (spawn ^onion-netlayer))))

;; Enliven the clients.
(define client1
  (with-vat vat
    (<- mycapn 'enliven (string->ocapn-id (second (command-line))))))
(define client2
  (with-vat vat
    (<- mycapn 'enliven (string->ocapn-id (third (command-line))))))

(define (^len bcom)
  (lambda (v)
    (on (all-of (<- client1 (first v))(<- client2 (second v)))
        (lambda (res)
          (sqrt (fold + 0 res)))
        #:promise? #t)))

(define len (with-vat vat (spawn ^len)))

(with-vat vat
  (let-on ((l ($ len '(3 4))))
    (format #t "~a\n" l)))

;; Wait for the result to be computed, otherwise nothing will be printed.
(sleep 3600)

The server also starts by connecting to the network, and then it registers (or “enlivens” in Goblins parlance) two clients by magic incantations. The IDs of the clients are supposed to be passed as string arguments through the commandline, which are retrieved by (second (command-line)) and (third (command-line)), respectively (as with argv in C, the first argument is the name of the program or Guile script itself, and unlike in C, counting starts with 1, not 0). So we obtain local variables client1 and client2. The remainder of the code is the same as in the serial example, except that we again combine the norm and square root computations into one function len. Finally we add a bit of waiting: This is necessary to wait for the resolution of the promises, since let-on does apparently not do so; otherwise the server script will terminate before the result of the computation is printed.

To run the example, do not forget to start the Tor daemon with the command

tor -f $HOME/.config/goblins/tor-config.txt

Then open three terminals, and in two of them launch a client with the command

guile client.scm

and copy the two URIs of the form ocapn://… In the third terminal, start the server with the command

guile server.scm ocapn://… ocapn://…

where the ocapn://… command line arguments are pasted from the client output. After a few seconds the server will print the result of the computation, and all three programs can be stopped using the <ctrl>-<c> key combination.

If nothing happens, chances are there is a problem with the Tor network; the file $HOME/.cache/goblins/tor/tor-log.txt may contain hints. In particular, the network needs to be 100% bootstrapped.

TCP instead of onions, after all

Even if used only locally, the need to access the Internet makes the Tor protocol relatively slow; connections can fail, and this makes debugging somewhat painful – it is not easy to distinguish a deadlock in the program code from a poorly working network. The Goblins documentation does not provide a working example for using TCP, but moving from Tor to TCP is relatively straightforward: Replace all occurrences of the substring onion in the scripts above (also in the name ^onion-netlayer and the symbol 'onion) by tcp-tls, then add the parameter "localhost" to the invocation of (spawn ^tcp-tls-netlayer). To simplify copying and pasting, here is the resulting code for client.scm:

(use-modules (srfi srfi-1)
             (goblins)
             (goblins ocapn ids)
             (goblins ocapn captp)
             (goblins ocapn netlayer tcp-tls))

(define vat (spawn-vat))
(define net (spawn-vat))

;; Define the client functionality.
(define (^square bcom)
  (lambda (x)
    (* x x)))
(define client
  (with-vat vat (spawn ^square)))

;; Helper function for printing IDs.
(define (print-id prefix id)
  (with-vat net
    (on id
      (lambda (sref)
        (format #t "~a ~a\n"
                   prefix (ocapn-id->string sref))))))

;; Create a communicator.
(define mycapn
  (with-vat net (spawn-mycapn (spawn ^tcp-tls-netlayer "localhost"))))
;; Create an ID for the client and print it.
(define id
  (with-vat net ($ mycapn 'register client 'tcp-tls)))
(print-id "Client ID" id)

;; Wait for requests.
(sleep 3600)

And server.scm becomes the following code:

(use-modules (srfi srfi-1)
             (goblins)
             (goblins actor-lib joiners)
             (goblins actor-lib let-on)
             (goblins ocapn ids)
             (goblins ocapn captp)
             (goblins ocapn netlayer tcp-tls))

(define vat (spawn-vat))
(define net (spawn-vat))

(define mycapn
   (with-vat net (spawn-mycapn (spawn ^tcp-tls-netlayer "localhost"))))

;; Enliven the clients.
(define client1
  (with-vat vat
    (<- mycapn 'enliven (string->ocapn-id (second (command-line))))))
(define client2
  (with-vat vat
    (<- mycapn 'enliven (string->ocapn-id (third (command-line))))))

(define (^len bcom)
  (lambda (v)
    (on (all-of (<- client1 (first v))(<- client2 (second v)))
        (lambda (res)
          (sqrt (fold + 0 res)))
        #:promise? #t)))

(define len (with-vat vat (spawn ^len)))

(with-vat vat
  (let-on ((l ($ len '(3 4))))
    (format #t "~a\n" l)))

;; Wait for the result to be computed, otherwise nothing will be printed.
(sleep 3600)

When starting the client, notice that the ID changes from a URI of the form ocapn://….onion/… to one of the form ocapn://….tcp-tls/…?host=localhost&port=…, where the port is chosen at random; the two clients and the server will each get their own port. (If desired, a given port can be chosen by adding a parameter such as #:port 12345 after "localhost" in the invocation of (spawn ^tcp-tls-netlayer).) Due to the special character & in the URI, it is necessary to enclose it in a pair of apostrophes ' on the command line, so one needs to start the server with the command

guile server.scm 'ocapn://…' 'ocapn://…'

That the parameter 'onion or 'tcp-tls is required in function calls such as ($ mycapn 'register client 'tcp-tls) is a surprising design choice in Goblins: When spawning the mycapn variable, a netlayer is passed as a parameter, so in theory the variable should be able to memorise the kind of network setting it is attached to.

Notice that with TCP, the result of the computation is printed immediately, whereas it takes a few seconds with Tor. So to ease debugging, we will from now on keep the TCP setting; going back to Tor is straightforward.

Registering clients

The approach in which the server needs to know all client IDs beforehand becomes unwieldy in a context where we expect hundreds or even thousands of computation cores. It would be preferable to use a two-stage process: The server publishes its ID, and the clients use it to connect to the server and to register their IDs. Then in a second step the server can send computing tasks to the clients. We will gradually transform the example code to end up with such a solution.

First of all, let us replace the fixed number (in our case, 2) of client variables by a more dynamic structure, a list of clients; for this, it is enough to modify the server as follows:

(define clients
  (with-vat vat
    (map (lambda (uri)
           (<- mycapn 'enliven (string->ocapn-id uri)))
         (list-tail (command-line) 1))))

(define (^len bcom)
  (lambda (v)
    (on (all-of* (map <- clients v))
        (lambda (res)
          (sqrt (fold + 0 res)))
        #:promise? #t)))

So here the client variable becomes a list instantiated using the (in principle variable number of) IDs passed on the command line. The variant all-of* of the joiner is used to treat lists of promises. Notice that <- can be used as any other function in a map statement: (map <- clients v) matches the two clients with the two entries of the vector and returns a list of promises resolving to the squares (for the time being we still assume that the length of the client list matches the length of the vector).

While we are at it, we may as well hold the list in a cell actor, as a way of introducing state by the backdoor: The cell may hold values that are exchanged throughout the program execution.

(use-modules (goblins actor-lib cell))
…
(define clients (with-vat vat (spawn ^cell '())))
(with-vat vat
  ($ clients (map (lambda (uri)
                    (<- mycapn 'enliven (string->ocapn-id uri)))
                  (list-tail (command-line) 1))))

(define (^len bcom)
  (lambda (v)
    (on (all-of* (map <- ($ clients) v))
        (lambda (res)
          (sqrt (fold + 0 res)))
        #:promise? #t)))

So instead of creating a list, we spawn a cell containing an empty list; then we put a different value into the cell by applying the $ function to it with the desired new value as additional argument. Later we extract the list by applying the $ function without additional argument to the cell (since we are in the same vat, we may use $ instead of <- and need not worry about promise resolution).

We are now prepared to implement the registration of clients in the server. For this, we create a new type of agent which takes a URI identifying a client and which adds it to the list of clients in the cell. To see that something actually happens, we then print the added URI:

(define (^register bcom)
  (lambda (uri)
    ($ clients (cons (<- mycapn 'enliven (string->ocapn-id uri))
                     ($ clients)))
    (format #t "Registered ~a\n" uri)))

We create an instance of this agent type, add it to the network and print its ID (using the same print-id function):

(define register (with-vat vat (spawn ^register)))
(let ((id (with-vat net ($ mycapn 'register register 'tcp-tls))))
  (print-id "Server ID" id))

Finally we can use this new register function instead of the ad-hoc creation to add the clients from the command line to the list:

(with-vat vat
  (map (lambda (uri)
         ($ register uri))
       (list-tail (command-line) 1)))

Altogether we arrive at the following code, which can replace the server.scm script while keeping the current clients:

(use-modules (srfi srfi-1)
             (goblins)
             (goblins actor-lib cell)
             (goblins actor-lib joiners)
             (goblins actor-lib let-on)
             (goblins ocapn ids)
             (goblins ocapn captp)
             (goblins ocapn netlayer tcp-tls))

(define vat (spawn-vat))
(define net (spawn-vat))

(define (print-id prefix id)
  (with-vat net
    (on id
      (lambda (sref)
        (format #t "~a ~a\n"
                   prefix (ocapn-id->string sref))))))

(define mycapn
   (with-vat net (spawn-mycapn (spawn ^tcp-tls-netlayer "localhost"))))

;; Register clients.
(define clients (with-vat vat (spawn ^cell '())))

(define (^register bcom)
  (lambda (uri)
    ($ clients (cons (<- mycapn 'enliven (string->ocapn-id uri))
                     ($ clients)))
    (format #t "Registered ~a\n" uri)))

(define register (with-vat vat (spawn ^register)))
(let ((id (with-vat net ($ mycapn 'register register 'tcp-tls))))
  (print-id "Server ID" id))

(with-vat vat
  (map (lambda (uri)
         ($ register uri))
       (list-tail (command-line) 1)))

;; Use clients.
(define (^len bcom)
  (lambda (v)
    (on (all-of* (map <- ($ clients) v))
        (lambda (res)
          (sqrt (fold + 0 res)))
        #:promise? #t)))

(define len (with-vat vat (spawn ^len)))

(with-vat vat
  (let-on ((l ($ len '(3 4))))
    (format #t "~a\n" l)))

(sleep 3600)

Now it is time to swap the roles! We first start the server without command line arguments (as it is written, it then just has an initial empty client list):

guile server.scm

Two clients are now started using the URI printed by the server as a command line argument:

guile client.scm 'ocapn://…'
guile client.scm 'ocapn://…'

For this to work, we need to add to the client script the necessary (and straightforward) code to enliven the server and to remotely register the client with the server. We end up with the following script client.scm:

(use-modules (srfi srfi-1)
             (goblins)
             (goblins ocapn ids)
             (goblins ocapn captp)
             (goblins ocapn netlayer tcp-tls))

(define vat (spawn-vat))
(define net (spawn-vat))

(define (^square bcom)
  (lambda (x)
    (* x x)))
(define client
  (with-vat vat (spawn ^square)))

(define (print-id prefix id)
  (with-vat net
    (on id
      (lambda (sref)
        (format #t "~a ~a\n"
                   prefix (ocapn-id->string sref))))))

(define mycapn
  (with-vat net (spawn-mycapn (spawn ^tcp-tls-netlayer "localhost"))))
(define id
  (with-vat net ($ mycapn 'register client 'tcp-tls)))
(print-id "Client ID" id)

;; Enliven server.
(define server
  (with-vat vat
    (<- mycapn 'enliven (string->ocapn-id (second (command-line))))))

;; Register with server.
(with-vat vat
  (on id
    (lambda (id)
      (<- server id))))

(sleep 3600)

Notice that we have slightly modified registration with the server: Since the client communicates directly with the server, there is no need to go through a string representation of the ID, which we may use directly as an argument to the function call. This assumes that in the server, registration has been modified as follows:

(define (^register bcom)
  (lambda (id)
    ($ clients (cons (<- mycapn 'enliven id)
                     ($ clients)))
    (print-id "Registered" id)))

Running the server and two copies of the client, one should now see the client IDs printed in their respective terminals, and messages in the server terminal that these clients have been registered. However, the desired length 5 is not printed. In fact, the len actor is called at the end of the server script before the clients have had a chance to register through the network (actually even before the clients are started), so the expression ($ clients) yields an empty list. Now the map function from SRFI 1 also truncates v to the empty list, all-of* resolves to the empty list, and fold returns the starting value 0, which is actually printed before the two client IDs.

This can be solved by having the server wait until the desired number of clients has registered, by adding the following code:

(define v '(3 4))
(while (not (eq? (length (with-vat vat ($ clients))) (length v)))
       (sleep 1))

As a warning, the equivalently looking lines

(define v '(3 4))
(with-vat vat
  (while (not (eq? (length ($ clients)) (length v)))
         (sleep 1)))

result in a deadlock in which none of the clients get a chance to register. It looks as if operations inside with-vat block the vat so that it does not handle incoming remote function calls.

After also removing the code that registers clients specified on the command line, the server script currently looks like this:

(use-modules (srfi srfi-1)
             (goblins)
             (goblins actor-lib cell)
             (goblins actor-lib joiners)
             (goblins actor-lib let-on)
             (goblins ocapn ids)
             (goblins ocapn captp)
             (goblins ocapn netlayer tcp-tls))

(define vat (spawn-vat))
(define net (spawn-vat))

(define (print-id prefix id)
  (with-vat net
    (on id
      (lambda (sref)
        (format #t "~a ~a\n"
                   prefix (ocapn-id->string sref))))))

(define mycapn
   (with-vat net (spawn-mycapn (spawn ^tcp-tls-netlayer "localhost"))))

;; Register clients.
(define clients (with-vat vat (spawn ^cell '())))

(define (^register bcom)
  (lambda (id)
    ($ clients (cons (<- mycapn 'enliven id)
                     ($ clients)))
    (print-id "Registered" id)))

(define register (with-vat vat (spawn ^register)))
(let ((id (with-vat net ($ mycapn 'register register 'tcp-tls))))
  (print-id "Server ID" id))

;; Use clients.
(define (^len bcom)
  (lambda (v)
    (on (all-of* (map <- ($ clients) v))
        (lambda (res)
          (sqrt (fold + 0 res)))
        #:promise? #t)))

(define len (with-vat vat (spawn ^len)))

;; Wait until enough clients have registered.
(define v '(3 4))
(while (not (eq? (length (with-vat vat ($ clients))) (length v)))
       (sleep 1))

(with-vat vat
  (let-on ((l ($ len '(3 4))))
    (format #t "~a\n" l)))

(sleep 3600)

Notice that the same code can be run for vectors with different numbers of entries; it just requires that (at least) as many clients connect as there are tasks to handle. As a small caveat, the code is correct as we did not implement an unregister procedure for the clients, so their number is monotonically increasing – otherwise it would be possible that between the arrival of the second client and the call to the len function, one of the clients has disappeared again and the clients list contains only one entry, say. Then the SRFI-1 map function we are using, which accepts lists of different lengths by truncating them all to the smallest occurring length, would only consider the first entry of v, and the incorrect length 3 would be computed.

In a more realistic setting, there are more computing tasks than clients. When these take all more or less the same time, they may be evenly split between the available clients. For instance, the following server code waits for two clients to connect and then computes the length of vectors of arbitrary dimension:

(use-modules (srfi srfi-1)
             (goblins)
             (goblins actor-lib cell)
             (goblins actor-lib joiners)
             (goblins actor-lib let-on)
             (goblins ocapn ids)
             (goblins ocapn captp)
             (goblins ocapn netlayer tcp-tls))

(define vat (spawn-vat))
(define net (spawn-vat))

(define (print-id prefix id)
  (with-vat net
    (on id
      (lambda (sref)
        (format #t "~a ~a\n"
                   prefix (ocapn-id->string sref))))))

(define mycapn
   (with-vat net (spawn-mycapn (spawn ^tcp-tls-netlayer "localhost"))))

;; Register clients.
(define clients (with-vat vat (spawn ^cell '())))

(define (^register bcom)
  (lambda (id)
    ($ clients (cons (<- mycapn 'enliven id)
                     ($ clients)))
    (print-id "Registered" id)))

(define register (with-vat vat (spawn ^register)))
(let ((id (with-vat net ($ mycapn 'register register 'tcp-tls))))
  (print-id "Server ID" id))

;; Use clients.
(define (^len bcom)
  (lambda (v)
    (on (all-of* (map <- ($ clients) v))
        (lambda (res)
          (sqrt (fold + 0 res)))
        #:promise? #t)))

(define len (with-vat vat (spawn ^len)))

(while (not (eq? (length (with-vat vat ($ clients))) 2))
       (sleep 1))

(define v '(1 2 3 4 5))
(with-vat vat
  (while (< (length ($ clients)) (length v))
     (let ((c ($ clients)))
       ($ clients (append c c)))))

(with-vat vat
  (let-on ((l ($ len v)))
    (format #t "~a\n" l)))

(sleep 3600)

The code somewhat crudely “doubles” the client list until there are at least as many occurrences of clients (with multiplicities) as tasks; then map does the right thing.

This simple situation occurs surprisingly often in number theory. For instance in ECPP, one needs to compute many modular square roots for the same modulus; trial factor many batches of numbers of the same size; do many primality tests for numbers of the same size. However, the more general case of tasks taking more or less long also occurs (in ECPP, for instance, when computing roots of class polynomials of vastly differing degrees). The relative task durations are also not necessarily easy to estimate. In a more distributed setting, one can also imagine that even homogeneous tasks are more or less quickly solved with more or less powerful participating machines. Scheduling tasks by hand is thus not realistic in general. Instead, one would need a more dynamic approach, in which the server maintains a list of tasks and a list of clients; whenever a client is idle it should be sent a new task.

Given the length of this second part, this is a question I plan to pursue in another instalment.

This blog post, originally published on 2024-09-04, was updated on 2025-02-25 to cover changes between Goblins 0.13.0 and 0.15.0 and to incorporate minor improvements.

Goblins for number theory, part 1

2024-09-04T00:00:00Z

Starting with Goblins

Motivation

Most of my code in algorithmic number theory is written in C and runs in a parallelised version using MPI on a cluster. The C language is mandatory for efficiency reasons; MPI is mostly a convenience. Indeed number theoretic code is often embarrassingly parallel. For instance my ECPP implementation for primality proving essentially consists of a number of for loops running one after the other. A server process distributes evaluations of the function inside the loop to the available clients, which take a few seconds or even minutes to report back their respective results. These are then handled by the server before entering the next loop. In a cluster environment with a shared file system, this could even be realised by starting a number of clients over SSH, starting a server, and then using numbered files to exchange function arguments and results, or touch on files to send “signals” between the server and the clients. Computation time is the bottleneck, communication is minimal, so even doing this over NFS is perfectly feasible (and I have written and deployed such code in the past). MPI then provides a convenience layer that makes the process look more professional, and also integrates more smoothly with the batch submission approach of computation clusters.

The very loosely coupled nature of number theoretic computations should make it possible to distribute them beyond a cluster. Why not even do a primality proof with several participants working together over the Internet? I have looked at BOINC previously; but the system seems to be intended for completely uncoupled problems, essentially exploration of a large search space. The work is cut up into a number of independent tasks that are sent out to the participants; if they do not report back in a few days, the same task is sent out again, and over several months all tasks are treated. While number theoretic computations may also take a few months, they do require at least some synchronisation, and the server needs to hear back from the clients every few minutes so as not to be blocked. (Each for loop is embarrassingly parallel, but several loops must be run sequentially.) Also the “administrative” overhead of things to be done for BOINC outside the program itself looks rather daunting: setting up a database, for instance. So I have been looking for a programming environment that is somewhere between MPI and BOINC, making loosely coupled computations possible; it should be able to run over the Internet and not only on a cluster connected by SSH; and it should result in code that is relatively easy to write, and for which just as with MPI the parallel version does not look very different from the sequential one. (In my C code, I usually end up having everything in one file, with the parallel and the sequential versions being handled by alternating blocks selected by #ifdef.)

Goblins is a distributed programming environment by the Spritely Institute that seems to fit the bill. It is meant for distributed programming, and locally running code can seamlessly be run over networks using various mechanisms such as Tor or simply TCP and TLS. On the other hand, not only does it use puzzling vocabulary, but also puzzling concepts, such as object “capabilities”, actor “model” and “vats”. For someone coming from imperative programming and good old C, with a penchant for assembly, looking at Goblins can feel like reading Heidegger. Fortunately Goblins comes with an excellent tutorial, which clarifies the seemingly exotic concepts by a hands-on approach with concrete examples. These put the emphasis, however, on the distribution and communication layer; tangible results are essentially obtained as side effects of printing values on screen. While I think it is an excellent idea to teach programming without mathematics to lower the barrier for people who do not like mathematics, I had the opposite problem: As a mathematician wanting to do computations, it was not immediately clear to me how to have the server send computational tasks to the clients and recover the results. Goblins is a library (or collection of “modules”) for Guile (or Racket), two Scheme dialects; from what I understand, it is unlikely that a C implementation will be available any time soon.

So the way in which I see Goblins being useful to distributed number theory computations is as follows:

Use Guile and Goblins to express the high-level control flow of the program, and additionally Goblins to express its parallelised and distributed aspects.
Use functions from a C library to do the heavy lifting, for instance functions from CM to do the computationally intensive tasks related to primality proving, which can be called from Guile using the Foreign Function Interface or FFI. In a number theoretic context, function arguments and values are often arbitrarily long integers coming from the GMP library. Given that Guile itself is written in C and relies on GMP for its implementation of integers, one can be hopeful that this should not pose too many problems.

The prototypical example

As a running example, I would like to treat the computation of the euclidean length of a vector; starting simple and enhancing the example step by step in the following tutorial, which I am making up while I am trying to solve the problem for myself.

Let us begin with a fixed vector of small, fixed size, which would look like the following in C:

int square (int x) {
   return x*x;
}

int v [] = {3, 4};
double len;

len = 0.0;
for (int i = 0; i < sizeof (v) / sizeof (int); i++)
   len += square (v [i]);
len = sqrt (len);

Granted, this it not number theory, but it fits the situation described in the motivational section above: There is a for loop going through the vector calling independently for each entry the square function, which stands for a function that would be expensive to compute, should be distributed to the clients and loaded from a C library; the additions and the square root, the + and sqrt functions, stand for cheap post-treatment done at the server level.

The following Guile code captures this sequential computation with the map and fold idiom in appreciable compactness:

(use-modules (srfi srfi-1))
(define (square x) (* x x))
(define v '(3 4))
(sqrt (fold + 0 (map square v)))

Open a Guile REPL using the guile command. Then copy-paste this code into the REPL; or save it as a file euclid.scm and type (load "euclid.scm") in the REPL; enjoy the Pythagorean result!

Before continuing, please go first through Chapters 1 to 4 of the Goblins documentation and tutorial.

The next step is to “locally distribute” the computation, that is, to create two clients and a server for the different steps of the computation; for the time being, these will all live in the same local Guile REPL. What is called a “process” in MPI corresponds to a “vat” in Goblins; so we create vat0 for the server and vat1 and vat2 for the clients:

(use-modules (goblins))
(define vat0 (spawn-vat))
(define vat1 (spawn-vat))
(define vat2 (spawn-vat))

Functions in an MPI instrumented parallel program correspond to “actors” living in a vat; we first define a type of actor computing squares:

(define (^square bcom)
  (lambda (x)
    (* x x)))

To distinguish it from the square function above, we prepend a ^ to its name; it takes a formal parameter called bcom that we need not worry about.

Then we populate the client vats with a square actor each and keep references to the different actors under different global names:

(define client1
  (with-vat vat1 (spawn ^square)))
(define client2
  (with-vat vat2 (spawn ^square)))

Now we can create a function in the server vat which computes the length of a 2-dimensional vector by calls to the client actors using <-. For this to work, we will need to wait for the clients to finish their computations (or, in Goblins parlance, for their “promises” to be “fulfilled”); this is done using on for each call to a client actor. The “Goblins standard library”, described in Chapter 6 of the documentation, comes in handy here; in particular we can use a joiner to wait for several actors at the same time.

(use-modules (srfi srfi-1)
             (goblins actor-lib joiners))
(define (^len bcom)
  (lambda (v)
    (on (all-of (<- client1 (first v))(<- client2 (second v)))
        (lambda (res)
          (let ((l (sqrt (fold + 0 res))))
            (format #t "~a\n" l))))))
(define len (with-vat vat0 (spawn ^len)))

The len actor can now be called as follows, which will print the euclidean length of a vector on screen; from within the vat where the actor resides, we may use $ instead of <-, which behaves like a normal function call:

(with-vat vat0 ($ len '(3 4)))

Putting this all together for convenient copy-pasting, here is the complete code:

(use-modules (srfi srfi-1)
             (goblins)
             (goblins actor-lib joiners))

(define vat0 (spawn-vat))
(define vat1 (spawn-vat))
(define vat2 (spawn-vat))

(define (^square bcom)
  (lambda (x)
    (* x x)))

(define client1
  (with-vat vat1 (spawn ^square)))
(define client2
  (with-vat vat2 (spawn ^square)))

(define (^len bcom)
  (lambda (v)
    (on (all-of (<- client1 (first v))(<- client2 (second v)))
        (lambda (res)
          (let ((l (sqrt (fold + 0 res))))
            (format #t "~a\n" l))))))

(define len (with-vat vat0 (spawn ^len)))

(with-vat vat0 ($ len '(3 4)))

Promises, promises!

The previous code prints the length of the vector on screen using the format function; one might wish to instead create a function that returns the value, to assign it to a variable for future treatment, for instance, or to enter the Guile equivalent of the next for loop. This turns out to be surprisingly difficult, or, to be more precise, impossible. The reason is that on handles the promise by calling the function in its body with the return value of the promise, but does not itself return the result of this evaluation, as I would have expected. (Promises in Guile itself, created with delay, behave in this expected way when using force.) It is possible to obtain a return value for on, but this will again be a promise and not a “real” value — once a promise, always a promise!

So instead of passing around values, one quickly ends up passing around promises; this requires to get used to, and entails an additional layer of wrapping everything into on and a function instead of just evaluating the body of the function. As far as I understand, to obtain any tangible result, one eventually needs to print it on screen or into a file. The following example illustrates how to use the #:promise? #t keyword parameter with on to ensure that it returns a promise, and how to lug this promise around to continue computations with its encapsulated value, while never leaving the realm of promises until eventually printing a result. It moves the computation of the square root out of the ^len actor.

(use-modules (srfi srfi-1)
             (goblins)
             (goblins actor-lib joiners))

(define vat0 (spawn-vat))
(define vat1 (spawn-vat))
(define vat2 (spawn-vat))

(define (^square bcom)
  (lambda (x)
    (* x x)))

(define client1
  (with-vat vat1 (spawn ^square)))
(define client2
  (with-vat vat2 (spawn ^square)))

(define (^norm bcom)
  (lambda (v)
    (on (all-of (<- client1 (first v))(<- client2 (second v)))
        (lambda (res)
          (fold + 0 res))
        #:promise? #t)))

(define norm (with-vat vat0 (spawn ^norm)))

(with-vat vat0
  (define n ($ norm '(3 4)))
  (define l (on n (lambda (x) (sqrt x)) #:promise? #t))
  (on l (lambda (x) (format #t "~a\n" x))))

So here, n and l are promises, and fulfillment occurs only in the last on that prints a result.

Now that we have understood how things work, it is useful to introduce the let*-on syntactic sugar, which lets us end up with the following code:

(use-modules (srfi srfi-1)
             (goblins)
             (goblins actor-lib joiners)
             (goblins actor-lib let-on))

(define vat0 (spawn-vat))
(define vat1 (spawn-vat))
(define vat2 (spawn-vat))

(define (^square bcom)
  (lambda (x)
    (* x x)))

(define client1
  (with-vat vat1 (spawn ^square)))
(define client2
  (with-vat vat2 (spawn ^square)))

(define (^norm bcom)
  (lambda (v)
    (on (all-of (<- client1 (first v))(<- client2 (second v)))
        (lambda (res)
          (fold + 0 res))
        #:promise? #t)))

(define norm (with-vat vat0 (spawn ^norm)))

(with-vat vat0
  (let*-on ((n ($ norm '(3 4)))
            (l (sqrt n)))
    (format #t "~a\n" l)))

This looks exactly like normal let* syntax in Guile! So in the end, we arrive at a program which looks as if it handled normal values, with all promises swept under the rug.

Acknowledgements

I thank Jessica Tallon and David Thompson for their kind help with understanding the concept of promises covered in the previous section.

This first part has dealt with basic programming concepts in Goblins. In the end, all our code still runs in a single script, so we have taken a twisted path to write essentially serial code, but in doing so, we have laid the groundwork for true parallelisation with Goblins.