The Wagner

Year 2025 in review

2025-12-31T00:00:00+01:00

The first quarter of the year I trained again for the 20 km of Lausanne where I beat my 2024 personal best. In May, I joined a few friends for a week of gravel cycling through the hills of Tuscany.

Articles

In February, I described our use of the AWS Cloud Development Kit at my new job. I believe I found a right length and format to explain why this tool helps me building cloud infrastructure.

In May, I published an anniversary post for my Homelab. For five years I manage my own home infrastructure with Nix which makes me proud and happy.

In October, I analyzed my note taking progress and explained how a two-step process helps me to write great notes. I also reviewed Charles Perrow’s Normal Accidents.

Finally, as in the past few years, in December I spent my free time solving the Advent of Code puzzles.

Tech

I spent some time exploring the NATS communication system and the Replicant library for building web applications. I hope I’ll have an opportunity to integrate them into future projects.

Books

Many years ago a friend of mine gifted me The Making of the Atomic Bomb by Richard Rhodes. I cannot recall why I waited so long to open it, I finally read it this summer. In just a few days I consumed Andy Weir’s Hail Mary Project. To finish the year, I spent my Christmas reading Bertrand Meyer’s Agile!.

Thanks for reading my blog and happy 2026!

Advent of Code 2025

2025-12-12T00:00:00+01:00

This year, again, December starts with the Advent of Code puzzles. Eric Wastl announced that from this year the puzzle series last only 12 days instead of the 25. In the previous years I felt exhausted by the end of the month so I don’t mind that this year ends earlier.

Puzzles

Between December 1 and December 12, a programming puzzle appears every day on the Advent of Code website. Each problem has two parts, the second part unlocks after you complete the first. To understand each day’s puzzle, go to the website and read the full description (for example Day 1: Secret Entrance).

Spoiler alert: If you still plan to work on the puzzles, stop reading now.

Highlights

For the sixth consequtive year, I write my solutions in Clojure. Since I don’t use this language professionally, I look forward to coding in Clojure every December. Usually, I solved the day’s problem before leaving to work, though sometimes I polished my code in the evening.

I sought external help to solve the second parts of Day 7 and Day 10 and I haven’t yet started Day 12. The other days went smoothly. I particularly enjoyed Day 6 which required reading the input file from right to left, to satisfy to the Elf’s clumsy specification.

You can find my solutions on GitHub.

Acknowledgments

Thanks Eric Wastl for creating and running Advent of Code.

Normal accidents

2025-10-26T00:00:00+02:00

This summer I read Normal Accidents by Charles Perrow, which analyzes failures in the nuclear industry, petrochemical plants, planes and airways, marine transport, and other earthbound systems like dams and mining. In the book’s title the word “normal” means systemic, accidents that involve the unanticipated interaction of many failures. Perrow argues that complex and tightly coupled systems with catastrophic potential have an inherent susceptibility to normal accidents.

No easy explanations

Perrow found that most accident analysis reports simplistically conclude that a unit failure, an operator error, or poor management caused an accident.

For instance, in Chapter 6 Marine Accidents, Perrow analyzes ship collisions, a large majority of which occur in inland waters in clear weather with a pilot on board. Furthermore, the two ships often do not initially follow a collision course; instead, one or both change course after becoming aware of the other in a manner that effects a collision. The crews had years of experience, the ships had all systems operational and neither ship intended to crash into the other.

Marine transport, including the ships, their crew and all the built-in safety devices form a complex system where interactions may occur in an unexpected sequence.

We reason about the behavior of a complex system via our own mental model. We use this model to draw conclusions about interactions which we cannot directly see. This helps us tremendously in finding problems when things go wrong. On the contrary, we may also reject signals from the system that contradict our own model of the world.

Instead of blaming a single component or a person, Perrow develops his DEPOSE framework which considers the Design, Equipment, Procedures, Operators, Supplies and materials, and the Environment of a system. The interactions among these aspects determine how a system behaves and how it fails.

System, subsystem, units and parts

Perrow uses a four-level hierarchical system model composed of parts, units, subsystems, and the full system. For example, a nuclear power plant as the system comprises a cooling subsystem which contains a steam generator unit with many parts like pumps, motors and piping.

I highlighted a unit that belongs to two subsystems, often called a “common-mode unit”. For example, if two subsystems use a single electricity source, a power outage may immediately disrupt both. Most systems contain such units, often for economic or efficiency reasons.

Most systems include safety features, often implemented as extra parts and units. Paradoxically, the addition of more safety mechanisms often increases the possibility of unanticipated interactions with existing components.

Interaction and Coupling

I reproduced Figure 3.1 (and later repeated as Figure 9.1) from the book, which organizes human and technological systems by the nature of interaction and coupling.

A system does useful work because its components interact with each other. Valves open and close to control the coolant’s flow. Public officials exchange ideas and documents in a government agency. A conveyor belt moves the car to the next assembly phase. By the nature of interaction Perrow arranges systems from linear to complex.

Perrow defines linear interactions as those that occur in some expected sequence. People can see and understand the cause and effect relationships in such interactions, even when they fail. As shown in the figure above, linear interactions dominate assembly-line production. In a car factory, when a workstation on a line malfunctions, parts pile up before failure point, but stations after keep working until they don’t receive new pieces anymore. Furthermore, a failure on one assembly line typically doesn’t affect neighbouring lines, at least not immediately.

On the right-hand side of the diagram we find complex systems with many hidden, hard-to-understand interaction patterns. Perrow spends an entire chapter on describing the Three Mile Island accident. He argues that operators couldn’t have possibly made better decisions during the accident because of the inherent complexity of a nuclear plant.

The vertical axis of the figure represents coupling. Tightly coupled systems have more time-dependent processes: they cannot wait or stand by until attended to. We cannot change resource quantities and qualities, we cannot easily substitute supplies, equipment or personnel. We cannot switch the order operations because often we only have one way to reach the production goal.

Perrow also applies his analysis to sociological systems like schools and R&D firms. For example, a junior college highly regulates the sequences of classes while at a university, students have more liberty to complete their required credits. By the book’s definition, this makes a junior college more tightly coupled than a university. A university doesn’t just teach, but also organizes research, collaboration with industry often involving politics. This makes a university, as a system, more complex than a junior college.

Fixing complex, tightly coupled systems

According to Perrow’s thesis, normal accidents occur in tightly coupled systems with complex interactions. Reducing either complexity or coupling would make such systems safer, but often the laws of nature prevent such changes. Most physical and chemical reactions only occur under specific conditions, requiring us to build control and safety systems around them.

If we cannot change a complex, tightly coupled technology — and we don’t want to abandon it —, perhaps we could prepare ourselves better to face unforeseen events.

In linear systems, where the unforeseen rarely occurs, designers rely on standards and regulations, operators complete training sessions to handle not only everyday operation but also failures. In other words, linear systems favor centralization.

Centralization also has beneficial effects on tightly coupled systems. With little buffer time, when a failure occurs the operator has no time to analyze the situation. Following a standard recovery procedure stops the failure from propagating to other parts of the system, and helps recovering normal operations.

Tightly coupled, complex systems pose incompatible demands during incidents. While tight coupling imposes unquestioned obedience and fast response from operators, the complex and incomprehensible nature of interactions requires a slow search from subsystem to subsystem to even recognize an ongoing incident.

Summary

Perrow wrote this book in 1984, inspired by the Three Mile Island accident five years earlier. The 1999 edition’s afterword contains a few extra references to books and papers that further developed Perrow’s ideas, now forming what we call Normal Accident Theory. He also added a few speculative sections about the risks associated with growing number of interconnected computer systems. Now we know that Y2K problem didn’t induce an apocalypse, but we definitely saw defective software causing catastrophes.

Though Normal Accidents contains many stories where technology went horribly wrong, Perrow puts humans at the center because we ultimately make the choices that help or harm others and the environment.

Acknowledgement

Thanks Rafał for suggesting and lending me this book.

My note-taking process

2025-10-11T00:00:00+02:00

I began taking notes around age ten in primary school. Though my teachers certainly instructed us to do so, I don’t remember if anyone actually taught me how to take notes. Like the rest of the class, I studied and revised using them. I followed the teacher’s explanations with a pen in my hand, jotting down words, sentences, and diagrams in my notebook. I maintained this habit during my entire studies. Still today, I prefer to follow a lecture while taking notes.

I view this in-class note-taking as a one-step process: I capture the teacher’s words on paper and the notes become directly usable for revising.

When I write at work, I produce a lot of notes. For example, during a meeting I capture the salient points of a discussion, but the resulting notes, in their raw form, only make sense to me and to none of my co-workers. A meeting possesses much less structure than a university lecture, the messy, somewhat unreadable in-meeting notes require an extra pass to render them usable.

When I read a technical text, I take notes. I don’t underline or highlight words in books; instead, I jot down the interesting word or concept together with the page number. Later, I go back to each notes’ page and form a complete sentence about the concept or idea.

A few years ago, reading How to Take Smart Notes by Sönke Ahrens made me think about my own note-taking process which now involves three kinds of notes:

Fleeting note: temporary reminder of an idea, thought, or piece of information.
Project note: information relevant to achieve a specific goal, often involving other people.
Permanent note: distilled ideas and arguments that form the core of a knowledge base.

The process still starts with the capture step, but it contains two more steps which produce the project and permanent notes:

Let’s see each step in detail.

Capture

During the capture step I write down any relevant information I want to hold on for later. I already mentioned writing meeting or literature notes; I jot down anything that switches on that light bulb above my head. The capture transforms an intangible idea into a fleeting note.

Since my school days, my capture method remained essentially the same. Depending on my mood I draft words, create lists, or sketch shapes and arrows. I don’t follow any particular system; I characterize my technique as a mix of linear and visual note-taking.

I haven’t seen two people take notes the same way so I suspect any method that produces legible written text yields usable fleeting notes. Wikipedia and many note-taking application vendors frame note-taking as this capture step.

Capturing fleeting notes became almost automatic to me, but this step alone doesn’t produce useful notes. I now believe the notes’ value only emerges when I do a second pass over my fleeting notes.

Who, What, When, Where, Why

The activity, which I name after the Five Ws of journalism, starts if a fleeting note belongs to a project. As David Allen explains in Getting Things Done you cannot do a project, you can only do actions that bring you closer to the project’s goals. When processing a project-relevant fleeting note, I identify actions and define the project’s next action.

In my personal system, I maintain a list of actions per project; the next action appears as the first element. At work, I store the project notes in whatever system the team chooses. The implementation doesn’t matter as long as the note reaches the people involved.

Rephrase

If a note doesn’t relate to a specific project, I write a permanent note and store it in my reference system. These notes come in different kinds including data sheets, manuals, literature review, definitions, and outlines.

For example, I keep a list of tools I require for replacing my bike’s break pads and a note detailing Git commands I use infrequently. I have notes about concepts and ideas from articles and conference talks, and I draft summaries of books I read.

Initially, I tried to segregate the different note kinds, but now I simply place every non-project related note in a single system.

Most fleeting notes demand work to reach the reference system. I rephrase the recorded concept using my own words. I write full sentences in one or two paragraphs, preferably in active voice, and I avoid lists and bullet points. For example, my permanent note about Paul Graham’s essay Writes and Write Nots reads as follows:

In his essay Writes and Write Nots Paul Graham, predicts that in couple of decades most people will loose their ability to write. Clear writing requires clear thinking which makes writing fundamentally hard. When you have a more prestigious job, you require to write more.

Today generative AI tools may write for you, but they don’t to the thinking part. Paul Graham’s prediction that the world will split into those who write and those who “write not”. Following his previous argument, this means we will have a split between “thinks” and “think nots”.

It took me only a few minutes to write these two paragraphs. I can immediately recall why this article resonated with me and explain it using my own words to other people if I want to.

I admit, not all my notes look like this; sometimes I cannot find better words than the original. Sometimes I just copy over my fleeting notes and move on, but when I skip the rephrase step I know I don’t get the most out my notes.

Summary

In the past few years I adopted a two-step note-taking process. I capture ideas as fleeting notes. For projects, I extract next actions from the fleeting note and record a permanent note in a system the project team can access. Otherwise, I rephrase the fleeting note using my own words to create a permanent note, which I store in a reference system.

Homelab turns five

2025-05-31T00:00:00+02:00

This day marks the fifth anniversary of my Homelab. Five years ago, as a COVID-19 lockdown project, I learned Nix and installed NixOS on my home computers.

While my Homelab may not compete in sophistication with setups you find on Reddit, I consider it a forerunner in longevity and maintainability. I attribute this success to the use of Nix, which I praised in many blog post before.

The current setup, which remained stable since 2020, doesn’t represent the first generation of my Homelab. I started to use Ansible back in 2015 because I had grown frustrated with the “install and forget” approach, where you configure a system and then quickly forget the details, making future updates difficult.

In 2015 I built a Gitolite server on Debian Linux. A year later I could control my Christmas lights using Home Assistant with the entire configuration represented as code. I also built a container host with LXC, an ownCloud instance and a media server.

In 2017, because Ansible worked reasonably well on my servers, I started writing a playbook for my primary working laptop. I wanted to codify my laptop’s configuration but I couldn’t build a reliable set of Ansible tasks and playbooks that worked independently of machine’s existing state. I would manually install new tools and struggled to document every change in Ansible. Perhaps I lacked the discipline or I insisted on the wrong approach; eventually I gave up. I never finished the configuration scripts for my laptop. The servers’ playbooks gradually stopped working because of the system updates and other out of band changes.

Over the years I kept looking for a better software configuration system. I explored alternatives such as cdist, fabric, and propellor, but none felt sufficiently different to Ansible. They often had smaller user communities and more quirks. I even used Salt professionally and followed mgmt for a time.

Finally, in 2020 I discovered Nix and I wrote my first Homelab post about it. I firmly believe that if your hobby or job involves managing servers, you should not use anything but Nix.

Happy birthday, Homelab!

Six months of CDK

2025-02-15T00:00:00+01:00

Last August I started a new job. I joined a team of engineers building cloud infrastructure using the Amazon Web Services (AWS) Cloud Development Kit (CDK) TypeScript library. I have had extensive experience building on AWS, but I haven’t used this tool professionally.

In this article I describe my experience with the CDK and I will argue that, if you can, you should prefer the CDK to any other tool when building cloud infrastructure on AWS.

Infrastructure as code

Cloud providers such as AWS expose thousands of management endpoints to programmatically interact with compute, storage and networking resources. In particular, AWS groups related endpoints into services, such as EC2, S3, just to name two out of more than 400 from their offering.

I remember learning AWS back in 2018: I felt swamped by all the three letter acronyms. I didn’t know if I needed EC2, ECS or EKS, or if EBS made more sense for an instance inside a VPC.

Initially, I studied the most commonly used services using the AWS web console used services. For example, I clicked on the “Launch new instance” button on the EC2 console, answered a series of questions — often just accepting the proposed default values —, then I watched the instance booting. I traced the details of the running instance to other resources to understand how they interact. This way I learned that running a single virtual machine instance requires a machine image, a launch template, a volume, a network, and a bunch of permissions.

While I appreciate the interactive and visual nature of the web console, especially for learning, I can’t build large systems by clicking around in my web browser.

I specify the system’s blueprint in plain text files, as source code, which a computer can translate into programmatic calls to the cloud provider’s API. I prefer this approach, commonly called infrastructure as code, because the blueprint mirrors the engineering intent and, using a version control system, I can track how the intent changes as the project evolves.

I still use the web console every day, but mainly in “read-only” mode. I rarely use it to create or change resources, but I inspect and study the resources the blueprint creates.

CloudFormation

AWS CloudFormation, announced in 2011, takes infrastructure blueprints, called templates, and provisions the required resources in the right sequence taking into account any dependencies between resources. CloudFormation can automatically deploy infrastructure into different regions using templates stored in Git repository.

Despite its advanced capabilities CloudFormation gathered a bad reputation. I used to judge it based on the syntax of its specification language: a verbose YAML configuration file sprinkled with awkward built-in functions.

I had to look beyond the syntax to realize that a CloudFormation template represents a simple data structure, built up from numbers, strings, lists and maps. You can generate, analyze and transform templates using any programming tool.

I believe AWS never wanted developers to write giant CloudFormation templates by hand, but it didn’t guide users what to do instead. AWS engineers, while working on an internal project, searched for a more expressive way of writing infrastructure specifications. Their ideas seeded the development of the CDK which AWS announced in 2019 as their official CloudFormation template generator.

Cloud Development Kit (CDK)

The AWS Cloud Development Kit (CDK) library, written in TypeScript, generates CloudFormation templates. AWS developed JSii, a technology to expose the TypeScript CDK modules to other popular programming languages such as Python, Go and Java to attract developers from all these communities. But, instead of the programming languages, I suggest studying the CDK’s programming model to generate CloudFormation templates.

Constructs

The Construct library forms the core of the CDK. The library has no dependencies, and it defines the Construct interface modeling a piece of system state. A construct may contain other constructs, forming a tree representing the infrastructure blueprint.

The CDK build process automatically generates large part of the CDK library from CloudFormation resource specifications. For example, the CfnBucket construct represents a mechanical translation of the AWS::S3::Bucket CloudFormation resource to TypeScript. The AWS documentation refers to these objects as Layer 1 constructs.

You rarely use these generated objects because CDK engineers also created Layer 2 constructs which equip Layer 1 constructs with reasonable defaults, convenience methods and other syntactic sugar. For example, the Layer 2 Bucket construct by default creates an S3 bucket that follows AWS’s security recommendations. Also, you can just write

bucket.grantRead(ec2-instance)

which creates the necessary IAM policies so that the EC2 instance can read objects from the bucket.

Finally, constructs modeling application specific patterns live at Layer 3. For example, CDK Pipelines construct library coordinates many AWS services to create a deployment pipeline for a CDK application. This library showed me the leverage the CDK offers: in just a few lines of code, I could create a continuous deployment pipeline that deploys my infrastructure into three regions using two AWS accounts.

I admit, the first few times I read the documentation, I didn’t pay close attention to this layering. When I develop with the CDK, I don’t have to think about this stratification: constructs at each layer present the same uniform interface, allowing me to freely combine any constructs I need. But, understanding the difference between these layers shaped my expectations towards specific constructs.

I view Layer 1 constructs as the platform’s primitive operations. Because every AWS service integrates with CloudFormation, often from the day of its public release, the corresponding Layer 1 construct quickly becomes available in the CDK via automatic translation process I mentioned earlier. In contrast, the handwritten Layer 2 constructs, if they exist, often introduce higher-level abstractions which may or may not work for your use case.

Take the VPC Layer 2 construct, for instance. It comes with generous defaults setting up two Availability Zones and NAT gateways in both. These make sense when you design for high availability, but feels like overkill if you just need to launch a few development machines. Or, sometimes you want to assign secondary IP addresses to instances running in a VPC, but this construct doesn’t allow for that; you’d have to build the missing bits using Layer 1 constructs.

Even with these caveats, I’ve found Layer 2 constructs generally effective in my projects. I always try to use them first before I consider Layer 1 constructs, or even raw CloudFormation.

Escape hatches

You can import existing CloudFormation templates, written in YAML or JSON, and treat them as CDK Constructs: I had used a small CloudFormation template in my project; it worked seamlessly, but after a few days I just rewrote it in TypeScript.

In some cases, you may find that a CloudFormation resource doesn’t cover all configuration options of a resource. In this case you can fall back to AWS SDK calls triggered on CloudFormation stack operation events such as Create, Delete, or Update. The CDK’s AwsCustomResource construct allows you to bridge any gap in the CloudFormation coverage.

Tooling

The CDK’s availability across many programming languages always intrigued me. Advocates often present this as a key benefit: a team developing a TypeScript application, for example, can now write its infrastructure definition in TypeScript as well. This eliminates the need to learn a new domain-specific language. An argument I’ll call “appeal to familiarity” and one I certainly don’t disagree with. But, looking again beyond the language syntax, I think leveraging an existing programming library ecosystem provides the most benefit.

When I started using the CDK I didn’t know TypeScript. I had used JavaScript and many other programming languages, so learning the basics of TypeScript didn’t pose a problem for me: the web community produced an incredible amount of learning material, libraries, package managers and other tools. After a few hours I already studied the “CDK world”, because the TypeScript scaffolding just worked.

Difficulties

I praised the CDK’s construct-based component model and its integration with other AWS tools and with existing programming language ecosystems. But during my six months journey I faced also a few difficulties which I describe next.

Moving resources

I like maintaining code which groups resource definitions into high-level, application-specific modules like “storage layer” or a “compute cell” because the structure provides me context about a particular resource’s role in the application. As I mentioned before, constructs provide a great modeling tool for building up these modules.

The CDK computes a resource’s unique identity based on its path within the construct tree. Hence, when you move a resource from one high-level construct to another, CloudFormation interprets this change as an instruction to recreate that resource under a new identity. Depending on the resource, this might cause service interruption, data loss, or both.

A technique I learned from the CDK documentation protects against accidental resource destruction: I add a unit test that asserts the stability of the critical resource’s logical identifier.

Using deferred values

In a CloudFormation template a Ref refers another resource’s property. During deployment, the CloudFormation service orders resource creation such that it can substitute the Ref with the property’s actual value.

The CDK models references using tokens. These tokens present as regular TypeScript strings, but their values use a special encoding. This design choice lets you write code that looks like direct property access, which the CDK translates into Refs when needed. On the flip slide, you lack a clear signal when you handle one of these deferred values.

Fortunately, most of the code I write avoids inspecting or manipulating tokens. If you develop a construct library I suggest studying how to check for unresolved tokens in your constructs.

EKS Blueprints

The final point I want to make doesn’t concern the CDK itself; it highlights a self-inflicted problem that can surface in any project. At work, we use the EKS Blueprints library which, to me, creates more issues than it solves. Instead of using plain constructs, this library layers a bespoke dependency injection system on top of the CDK’s existing construct programming model. Also, this implementation heavily relies on async/await, which makes escaping its patterns incredibly difficult.

Summary

If you build on AWS, I suggest using the Cloud Development Kit. You may have reasons to choose something else, but I believe you should consider the CDK first before anything else. For learning, I recommend the official documentation, and especially the best practices section which offers many useful tips on structuring your AWS accounts, deployment pipelines, and CDK code. Finally, use any constructs from the CDK, but think twice before importing other construct libraries.

Year 2024 in review

2024-12-31T00:00:00+01:00

I started the 2024 with an intense training program for the 20 km of Lausanne. I moved to a new place, I rode my new bike as much as I could and I started a new job. Fortunately, next to all these changes in my life, I could still find some time for writing.

Articles

In April, I reviewed The Stupidity Paradox, a book exploring the role of functional stupidity in contemporary organizations. I liked writing this article because it helped me to internalize the content of the book.

At work, I heard many complaints about software quality, but nobody could define what quality meant for them. I decided to review the chapters on quality from Facts and Fallacies of Software Engineering. When I shared this post with a friend, he immediately suggested me to read The Zen and Art of Motorcycle Maintenance. In this book the protagonist becomes insane during his quest to define quality. In the article I cite seven attributes which form a working definition of quality for software teams without risking madness.

In the summer, I had some time between jobs to tinker in my Homelab. In this article I explain how I use my Raspberry Pi Camera module with NixOS.

Finally, as in the past few years, in December I spent my free time solving the Advent of Code puzzles.

Books

I read Liu Cixin’s Remembrance of Earth’s Past series in English: The Three-Body Problem, The Dark Forest and Death’s End. I found these books less captivating than the Expanse, the last big science fiction series I had read. Instead, I enjoyed watching the Netflix TV adaptation.

During vacation, I found Jack London’s Martin Eden on a public bookshelf. A random pick that became my best read of the year: after a thousand pages of science fiction the story of a struggling autodidact writer came as a breath of fresh air.

Thanks for reading my blog and happy 2025!

Advent of Code 2024

2024-12-25T00:00:00+01:00

Since 2020, every year I solve the Advent of Code puzzles. Also, every year I write an experience report about my 25+ day journey in December. Read my thoughts about previous years here: 2020, 2021, 2022 and 2023.

Spoiler alert: If you still work or plan to work on the puzzles, stop reading now.

Puzzles

Between December 1 and December 25, a programming puzzle appears every day on the Advent of Code website. Each problem has two parts, the second part unlocks after you complete the first. To understand each day’s puzzle, go to the website and read the full description (for example Day 1: Historian Hysteria).

Setup

My setup and goal remain the same as in the previous years: I use Clojure and I try to write succinct and comprehensible programs.

I don’t have a specific library of helper functions for the Advent of Code, but I have a best-first search implementation that I find useful in many problems. This year — in particular in days 16, 18, and 20 — I expressed the solution using this function.

My Day 16, Day 22 and Day 23 solutions run for about a minute, but I didn’t have the energy to optimize them.

Highlights

The first difficulty arrived on Day 12. The puzzle asked for counting the straight edges of polygonal “gardens”. Initially I didn’t find a good approach for identifying a garden’s boundary. When I started to track the normal of each point that form the boundary, my solution started to make sense.

I solved Day 13 with pen and paper then I implemented the resulting formula in code.

Day 14 had a picture of a Christmas tree hidden in the solution, so I wrote:

(defn solve-part2 [input]
  (first (🎄 input)))

Day 15 reminded me of Sokoban, one of the first games I played on a computer. I enjoyed this puzzle even if the second part of took me a few attempts to get it right. I experimented with using Clojure’s dynamic bindings to write code that solve both parts of the puzzle.

Debugging the assembly code in the second part of Day 17 went much better than in a similar problem in 2021. I worked on this problem together with a colleague which made it even more fun.

I couldn’t solve the second part of Day 21. The puzzle asked to find a shortest command sequence to control a robot pressing buttons on a numeric keypad using a series of directional keypads. I keep thinking about this problem, but so far I didn’t have the motivation to go back to it.

On Day 24 I didn’t write any code for the second part. The puzzle asked for finding four incorrectly wired logic gates of a full adder. Using Graphviz, I generated a diagram from the puzzle input and I looked for nodes which caused visible irregularities.

Summary

I could solve all but one problem during the 2024 Advent of Code: my best year so far.

You can read the code of my solutions on GitHub.

Acknowledgments

Thanks Eric Wastl for creating and running Advent of Code.

NixOS on the Lenovo Carbon X1 (Gen 12)

2024-09-10T00:00:00+02:00

My new Lenovo X1 Carbon Gen 12 arrived today. I plan to use this computer as my personal workstation for the coming years. I’ve been watching the evolution of the X1 Carbon series for a while; I was convinced that this is the right model for me. I don’t need extreme performance and I appreciate its small size and light structure.

The new X1 replaces a Lenovo X230 which I used for twelve years. The hardware is still in a good shape and the battery lasts for a few hours. But it’s just too slow even for browsing the Internet.

I run NixOS on every computer I own and the new X1 is no exception. In a few minutes I could replicate my current configuration on the new laptop. On the X1 I decided to use ZFS, a file system I like to learn more about. Also, the disk partitioning scheme is now defined declaratively using disko.

When I first booted NixOS 24.05 with its default 6.6 LTS kernel, the HDMI port and the sound didn’t work. I tried a few more recent kernel versions and I found that since version 6.8.12 everything works. Currently my system runs 6.10.2.

Acknowledgement

Thanks Kornél for helping me with the Lenovo hardware specifications, disko and ZFS.

Raspberry Pi camera on NixOS

2024-07-31T00:00:00+02:00

In this article, I describe how I configured my Raspberry Pi v1 camera module on my Raspberry Pi 3 running NixOS.

The Raspberry Pi OS has excellent support for many camera modules. If you run the officially supported operating system, most cameras work without any further configuration.

In my Homelab I don’t use the Raspberry Pi OS, but I run NixOS on all my computers, including the Raspberry Pi. NixOS works well on the Pi, but the camera module I own needs a special configuration.

The “new” camera stack

Four years ago, the Raspberry Pi team released a new camera stack that provides better access to the internals of the camera system. Today, the old stack using Broadcom proprietary software is unsupported and obsolete. Unfortunately, many online instructions and tutorials still refer to the Broadcom stack which renders them irrelevant to the modern camera stack.

The new camera stack comprises four layers:

Linux kernel with board-specific configuration
Camera-specific drivers
The libcamera library
rpicam-apps camera utilities for taking photos and videos

The next sections describe how I configured these components on NixOS to use a Raspberry Pi v1 camera module (Omnivision OV5647) with my Raspberry Pi 3B.

Kernel

The official NixOS installer works well on the Raspberry Pi. However, the camera modules require some subsystems that are not enabled by default.

Fortunately, it’s easy to switch to a kernel tailored to the Raspberry Pi. For my model 3B, I select the linux_rp3 kernel package:

{ pkgs, ...}:
{
  boot.kernelPackages = pkgs.linuxKernel.packages.linux_rpi3;
}

Actually, I don’t adjust this parameter in my configuration, but I import the raspberry-pi-3 module from nixos-hardware which selects the correct kernel and tunes a few other parameters too.

Camera driver (OV5647)

The Raspberry Pi v1 camera module uses an Omnivision OV5647 image sensor. We need to describe the OV5647 hardware to the kernel so that the sensor can be controlled via the respective system calls. The data structure and language for describing hardware is called the device tree.

This is NixOS configuration block that enables the image sensor:

{ pkgs, ...}:
{
  hardware.deviceTree.filter = "bcm2837-rpi-3*";
  hardware.deviceTree.overlays = [
    name = "ov5647-overlay";
    dtsText = '''
       ...ELIDED...
    ''';
  ]
}

For brevity, I omitted the value of the dtsText. In my Homelab repository you can read the full device tree overlay configuration.

The dtsText string is a copy of the file ov5467-overlay.dts from the Linux kernel source with one modification: I changed the compatible property from bcm2835 to bcm2837. It turns out the OV546 device tree overlay is compatible with both BCM chipsets, but I don’t understand why overlay’s source doesn’t reflect this.

I learned about this technique in a GitHub issue comment. It works™, the kernel recognizes the image sensor on the I2C bus:

$ cat /sys/bus/i2c/devices/10-0036/name
ov5647

It was hard to get the device tree overlay working, but I’m not entirely satisfied: fiddling with the device tree source files doesn’t feel right.

Fortunately, the hardware configuration is done. Next, the software stack.

libcamera

The libcamera library drives the Raspberry Pi’s camera system directly from the Linux kernel, with minimal proprietary code running on the Broadcom GPU.

libcamera is part of nixpkgs, but it’s built without support for the Raspberry Pi. I developed a Nix package overlay, based on the Raspberry Pi specific libcamera instructions, which the compiles libcamera with a few additional flags:

mesonFlags = old.mesonFlags ++ [
  "-Dcam=disabled"
  "-Dgstreamer=disabled"
  "-Dipas=rpi/vc4,rpi/pisp"
  "-Dpipelines=rpi/vc4,rpi/pisp"
];

The upstream libcamera package uses the Meson build system. In the previous snippet, old.mesonFlags refers to the upstream package’s build flags. The overlay appends to the original flag list enabling the Raspberry Pi specific Image Processing Algorithms (IPAs) and pipelines.

This example shows off the strength of the Nix Packages overlay systems: the overlay describes the differences from the upstream package’s build instructions, like a patch representing the differences between two versions of a text file.

rpicam-apps

rpicam-apps is a set of command line applications, built on top of libcamera, to capture images and video from a Raspberry Pi camera.

Based on a draft pull-request in nixpkgs, I wrote a Nix derivation which builds rpicam-apps using my own libcamera build and enabling only a minimal set of features. For example, I deactivate support for preview windows and advanced post-processing capabilities.

Picture time

I collected the configuration I described in the previous sections in a NixOS module which is imported from my Raspberry Pi 3 host configuration.

After the configuration is deployed on the device, I can list the available cameras:

$ rpicam-still --list-cameras
Available cameras
-----------------
0 : ov5647 [2592x1944 10-bit GBRG] (/base/soc/i2c0mux/i2c@1/ov5647@36)
    Modes: 'SGBRG10_CSI2P' : 640x480 [58.92 fps - (16, 0)/2560x1920 crop]
                             1296x972 [43.25 fps - (0, 0)/2592x1944 crop]
                             1920x1080 [30.62 fps - (348, 434)/1928x1080 crop]
                             2592x1944 [15.63 fps - (0, 0)/2592x1944 crop]

And, I can take a picture:

rpicam-still --output test-image.jpg

I won’t try to impress you with the quality of the recorded image. The v1 camera module is old, the modules available today have much better sensors and optics.

Summary

I use NixOS because the operating system is described in a declarative configuration. The camera specific modifications I described in this article are in a NixOS module which is included in my Raspberry Pi’s host configuration.

By choosing NixOS over the official Raspberry Pi OS I set myself up to an arduous journey. But, I learned about details of the camera stack, and now I appreciate more the integration work done by the Raspberry Pi Foundation.

Software Quality

2024-06-25T00:00:00+02:00

I was always part of software teams that wanted to write high quality software. Most team members felt what was good or bad quality software. Some goodness criteria were accepted by everybody in the team, perhaps some remained controversial.

I’m sure every team had heated discussions about software quality. Yet, it’s surprisingly hard to find a working definition of it. In this post I review the definition of software quality found in the book Facts and Fallacies of Software Engineering by Robert L. Glass from 2002.

Definition

Fact 46 of Facts and Fallacies of Software Engineering defines software quality as a collection of the following seven attributes:

Reliability is about a software product that does what it’s supposed to do, and does it dependably.
Usability is about the ease and comfort of use.
Understandability is about a software product that is easy for a maintainer to comprehend.
Modifiability is about software that is easy for a maintainer to add new capabilities without breaking existing ones.
Efficiency is about the economy in running time and space consumption.
Testability is about a software product that is easy to test.
Portability is about a software product that is easy to move to another platform.

There is no general, correct order of these attributes. The priorities of each software project are different.

Discussion

According to Glass, this definition of quality took the test of time, yet it remains controversial. Glass warns against attaching unrelated attributes — development cost, user satisfaction or delivering on time — to software quality. Of course it’s desirable if a user receives the software on time and they are satisfied with it. This may be due to good customer support and excellent project management. User satisfaction is certainly influenced by software quality, but not defined by it.

The definition of quality is industry specific. Just like the priority of the quality attributes is project specific.

Reliability and efficiency can be measured or estimated, for example, by monitoring tools. The other attributes are fuzzy and subjective which makes it impossible to express quality using “hard” metrics. This is not a problem because measurement is not invaluable for managing quality. Humans can manage research, design and many intellectual and creative things without numbers guiding them. Accepting that some attributes remain qualitative is better than pretending that a chart with made-up data is anyhow relevant.

Responsibility

Managers are responsible for the success of a software project. But quality is not a management job. Each of the quality “-abilities” have deeply technical aspects, therefore it is the responsibility of technical team to work towards the desired quality level.

Managers do, however, have an important role to establish a culture where achieving quality is given high priority. They can hire good engineers and let them build great software.

Summary

Software quality is a collection of seven attributes: reliability, usability, modifiability, efficiency, testability and portability. Not all attributes make sense in all situations so teams should make their own priority list of these attributes for each project. Ensuring quality is a technical job. It’s impossible to measure software quality, but this doesn’t mean it cannot be managed.

The Stupidity Paradox

2024-04-19T00:00:00+02:00

This is a review of the book The Stupidity Paradox by Mats Alvesson and André Spicer which explores the role of functional stupidity in contemporary organizations.

The authors’ thesis is that functional stupidity is omnipresent, especially in large firms, and it has a mix of positive and negative outcomes. The paradox is that presence of stupidity is not always bad, but it can have some benefits.

Stupidity Today

In the first part of the book the authors claim that predictions from the 1960s about the arrival of knowledge-based economy still remains a promise. The reality is that only a small fraction of jobs are knowledge-intensive. In fact, contemporary organizations make many smart people do stupid things.

Functional stupidity is the absence of reflection on the purpose or the wider context of a job. You do the job correctly, focusing on the technical details but stop searching for questions about the work. Three aspects characterize functional stupidity:

Lack of reflexivity: You don’t think about your assumptions.
Lack of justification: You don’t ask why you’re doing something.
Lack of substantive reasoning: You don’t consider the consequences or wider meaning of your actions.

Wishful thinking, following leaders without scrutiny, unreasoning zeal for fads and fashions, senseless imitation of others and the use of clichés in place of careful analysis are examples of functional stupidity.

In organizations a small amount of functional stupidity is beneficial. Avoiding difficult conversations can help individuals to suppress their doubts, be happy and feel comfortable with ambiguity. Ignoring negative impulses help to get along better with colleagues and provide a steady climb on the corporate ladder.

But if people stop asking probing questions and ignore problems for too long, functional stupidity can lead to larger problems and disasters. People grow cynical and alienated when they see a large discrepancy between proclaimed values and actual work.

Five kinds of Functional Stupidity

The second part of the book explores five sources of functional stupidity which are common in organizations, induced by leadership, structure, imitation, branding and culture.

Leadership-induced stupidity. When people develop an unquestioning faith in their boss and in the magical powers of leadership.

Structure-induced stupidity. Formal processes and structures are required in organizations but they don’t guarantee quality, reliability and productivity. People often blindly trust processes and systems which don’t produce the results they hope for. The mixture of senior managers who mainly sit in meetings talking to other managers, narrowly focused experts, and routinized workers create organizations where rule-following trumps good results.

Imitation-induced stupidity. Managers often adopt structures and formal practices that look good and not what makes the organization function more efficiently. This is to create an image which conforms to broadly shared expectations of how an organization should be. But a disconnect between the organization’s image and its everyday practices lead to frustration, low commitment and cynicism.

Branding-induced stupidity. There is a massive overproduction in many parts of the economy. To dispose of this surplus we developed an economy of persuasion. Branding helps to transform the dull job of convincing people about things they don’t need or want into something that sounds exciting and interesting. But branding activities are often met with indifference and cynicism.

Culture-induced stupidity. Corporate culture coordinates people, offers a shared sense of purpose, creates a common identity and reduces conflicts and confusion. But culture always includes a degree of functional stupidity. Most organizations foster a culture of optimism, focusing on the present and being change oriented. But when you can’t mention bad news you often can’t adapt to important changes. If you don’t look at the past you cannot learn from it. And an organization that drifts from one change initiative to another without real benefits becomes self-obsessed and gets harder to get actual work done.

Stupidity management

Although it’s never presented explicitly, stupidity management, that is to reduce thinking at work, is an important activity for managers. Again, small amount of functional stupidity can be advantageous because too much thinking can lead to conflicts, uncertainties, doubts and reduced motivation.

The third part enumerates four ways managers encourage functional stupidity:

Authority: Manager use their formal position in the hierarchy to make subordinates follow polices and orders. The hope for a reward or the fear of punishment discourages staff from thinking too much.
Seduction: Managers enlist attractive ideas and arrangements to persuade people. It’s often about arguing that change is always good and about painting a rosy picture of the future absent of the past and present problems.
Naturalisation: Managers claim that the organization’s assumptions, its view of the word, and goals are self-evident, or natural.
Opportunism: Managers buy into questionable trends or actions because good things will follow. Instead of searching or explaining the purpose of an action they rationalize the doubts.

The closing chapter proposes to fight functional stupidity with critical thinking: querying assumptions, asking for and being prepared to give justification and considering the outcomes and meaning of what we do.

The authors suggest that organizations instead of encouraging only optimism they should build up negative capability, the ability to face uncertainty, paradoxes and ambiguities. Some steps in this directions are:

Post- or pre-mortems: Take a good look at a failed project and learn from it. Or, before a project starts try to imagine all possible ways the project could fail.
Listening to newcomers and outsiders: New employees or an external person may point out deficiencies or silly things in your organization that you’ve grown accustomed to.
Reflective routines and anti-stupidity task force: Regularly stop to think and ask: Why?. Evaluate your projects, structures and processes. Do they make sense? What’s their purpose? Perhaps even try to discontinue or cancel an activity or an arrangement.

Summary

I’d recommend The Stupidity Paradox to everybody who works for any organization that employs more than a dozen people. Functional stupidity — characterized by the lack of reflexivity, lack of justification and the lack of substantive reasoning — is everywhere in today’s organizations. This sounds depressing. Indeed, even the authors are concerned:

Everyone from CEOs to low-level employees is regularly put at risk of overdosing on stupidity management. We believe that this corporate no-think is one of the most urgent, yet most challenging, issues that organizations face today.

But the book also makes the point that inhibiting individual thinking in certain cases results in less conflicts, happier work environment and higher productivity.

The Stupidity Paradox helps to spot elements of functional stupidity and provides some advise how to keep it under control. In summary, the antidote of functional stupidity is critical thinking: reflecting on your assumptions, on your actions and on their consequences.

Year 2023 in review

2023-12-31T00:00:00+01:00

This post is a short summary of the articles I wrote in 2023.

At the beginning of the year reviewed Winston Royce’s 1970 paper on Managing the development of large software systems. This paper is list of suggestions about what makes a software project succeed. Perhaps at places the language is archaic, but most of Royce’s observations still apply today.

I spent a few hours every month to improve my Homelab. I started hacking on some spare computers at home in 2019, because I was interested in configuration management systems such as Ansible, Salt and Puppet. Then, I discovered Nix which became the most important tool to build all my projects and to configure my servers.

This year I configured automatic deployment for all my servers at home. This includes a Raspberry Pi for which building software using freely available build servers such as GitHub Actions is non-trivial.

Finally, as in the past few years, in December I spent my free time solving the Advent of Code puzzles.

Thanks for reading and happy 2024!

Advent of Code 2023

2023-12-25T00:00:00+01:00

Since 2020, every year I solve the Advent of Code puzzles. This post, like those of previous years, is an experience report of my 25+ day journey in December.

Spoiler alert: If you’re still working or planning to work on the puzzles stop reading now.

Puzzles

Setup

This year again, now the third time, I wrote my solutions in Clojure. Every year I understand more of the standard library and I feel I write more concise and more idiomatic code than in the previous years.

Usually I solved the day’s problem before going to work, sometimes I coded a bit in the evening to polish my solution.

My Day 14 and Day 22 solutions are slow: they run for about a hundred seconds to find the solution. I suspect that these programs would be more efficient if I wrote them as a loop updating some local mutable state. Next year, I want to study profiling and optimizing Clojure code.

Highlights

On Day 12 I was proud to find a recursive solution using dynamic programming. In previous years, for example Day 10 in 2020, I struggled with similar problems.

I solved Day 14 and Day 20 by finding cycles in a periodic solution. This is also something I was more comfortable this year than before.

It was difficult to wrap my head around the interval arithmetics required to solve Day 5 and Day 19. Finally, I brute forced Day 5 and put aside the second part of Day 19 for a few days.

The second parts of Day 18, Day 21, Day 23 and Day 24 were the hardest to crack. I couldn’t solve these problems on my own and I used some help from Reddit.

On Day 18 I was on the right track, but I didn’t figure out how to apply Pick’s theorem to find the solution.

Day 21 was just too hard for me: I had some ideas, but nothing worked. I failed to recognize the hidden patterns in the input data. A typical Advent of Code puzzle.

Day 23 asked to find the longest path in a labyrinth. This is, as I learned this year, a much harder problem than finding the shortest path. I understood how to reduce the size of the search space, but my code is messy and I made a lot of mistakes during the implementation.

Day 24 was about line intersections. I understand well the underlying mathematical models but I failed to find a set of linear equations that yielded the solution. During the fourth week, I was tired and impatient to do symbolic math.

Day 25 asked to remove three vertices from a graph to create two isolated groups of nodes. Instead of writing code, I solved this problem graphically: I visualized the graph with the Graphviz and manually selected the nodes to remove.

Summary

2023 Advent of Code started well and I had a few small victories on some harder days. The second parts of Day 18, Day 21, Day 23 and Day 24 were too hard for me and I used external help. As always, on these tough days I learned the most.

As usual, after solving the puzzles I like to see how others approached the same problem. Here are some repositories I regularly checked:

My solutions are available on GitHub.

Acknowledgments

Thanks Eric Wastl for creating and running Advent of Code.

I enjoyed exchanging ideas and analyzing solutions with my friend Kornél.

Homelab deployment

2023-11-25T00:00:00+01:00

I configured continuous deployment in my homelab. This article describes how I use Nix with GitHub Actions and Cachix Deploy to automatically deploy NixOS machines on my home network.

Overview

My home network comprises a wireless router, a few computers, temperature and humidity sensors and remote controllable switches. I configure these devices mainly using Nix, and when it’s possible, I run NixOS on them. I use this setup to experiment with home automation and to learn about new tools. The entire configuration of my home network is available on GitHub.

The configuration of the NixOS servers was always built from a declarative specification stored in version control system. But the deployment of the new configuration was manual, slow and often tedious. I ran nixos-build from the command line, and sometimes I had to wait an hour to build and deploy the new machine configuration.

I wanted to reconfigure my servers automatically, triggered by committing to a Git repository.

To solve this problem I built a system based on Nix and freely available hosted services:

The automatic deployments work as follows:

The Homelab repository contains the NixOS host configurations.
A workflow in GitHub Action builds the NixOS system and stores it in a hosted binary cache.
An agent on the target machine pulls the built binaries from the cache and activates the new deployment.

Host configuration

The configuration of my servers is written in the Nix language. For example, host-nuc.nix describes the hardware and software configuration of my Intel NUC device:

{ config, ... }:

{
  imports = [
    ./hardware/nuc.nix
    ./modules/cachix.nix
    ./modules/common.nix
    ./modules/consul/server.nix
    ./modules/git.nix
    ./modules/grafana
    ./modules/loki.nix
    ./modules/mqtt.nix
    ./modules/prometheus.nix
    ./modules/push-notifications.nix
    ./modules/remote-builder
    ./modules/traefik.nix
    ./modules/vpn.nix
  ];

  system.stateVersion = "22.05";
}

The configuration is split into modules. Glancing at the imports block you can tell what is installed on this machine. The host nuc is my main home server, it runs all my “production” services.

For example, the cachix.nix module configures the Cachix Agent which is responsible for reconfiguring the NixOS machine when a new build is available. The agent runs on all machines whose configuration automatically managed.

Building with GitHub Actions

The NixOS host specifications are built with a single nix build command:

nix build .#nixosConfigurations.nuc.config.system.build.toplevel

This builds the whole system for the nuc machine: the kernel, the installed packages and their configuration. By default, nix downloads pre-built packages from the NixOS public binary cache, so a typical execution of the build command takes only a few minutes.

To run the build in GitHub Actions, the build job installs Nix and configures the access to the Cachix binary cache where the deployment artifacts are stored:

jobs:
  build:
    runs-on: ubuntu-latest
    environment:
      name: Homelab
      url: "https://app.cachix.org/deploy/workspace/lab.thewagner.home/"  # ⑴
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-qemu-action@v3                                 # ⑵
      - uses: cachix/install-nix-action@v23
        with:
          extra_nix_config: "extra-platforms = aarch64-linux"             # ⑶
      - uses: cachix/cachix-action@v12                                    # ⑷
        with:
          name: wagdav
          authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
      - run: nix build --print-build-logs .#cachix-deploy-spec            # ⑸
      - run: |                                                            # ⑹
          cachix push wagdav ./result
          cachix deploy activate --async ./result
        env:
          CACHIX_ACTIVATE_TOKEN: "${{ secrets.CACHIX_ACTIVATE_TOKEN }}"

The build job of the workflow:

Configures a deployment target. When a GitHub Actions workflow deploys to an environment, the environment is displayed on the main page of the repository.
Installs the QEMU static binaries for building packages for architectures different than that of the build runner.
Configures Nix to use emulation to build ARM 64 packages.
Configures the Cachix hosted binary cache.
Builds the deploy specification which is a set of the NixOS systems to deploy.
Pushes the built binaries to the cache and sends an activation signal to the Cachix Agent.

The job uses two secrets: CACHIX_AUTH_TOKEN is the authentication token to push to the binary cache and CACHIX_ACTIVATE_TOKEN is required to activate the built NixOS configurations.

Deploying with Cachix Deploy

To deploy the built NixOS configurations I use the generous free-tier of Cachix Deploy. Following their documentation, I installed cachix-agent on the target hosts and configured a few authentication keys.

The agent process connects to the Cachix backend and waits for a deployment. When a new deployment is available, the agent pulls the relevant binaries from the binary cache and reconfigures the NixOS system it runs on.

Summary

Since I configured automatic deployment of this blog I wanted the same for my home infrastructure. I had my configuration repository and I knew how to build the machine configurations with GitHub Actions. I was missing the automatic deployment part until Cachix Deploy was announced. Cachix has excellent documentation and the integration with my Homelab was super simple.

Acknowledgement

I’m grateful to Cachix Deploy for offering a binary cache and a deployment service.

Building Nix packages for the Raspberry Pi with GitHub Actions

2023-11-20T00:00:00+01:00

Building Nix packages for the Raspberry Pi 3 or newer requires building for an ARM 64 architecture, which Nix refers to as aarch64-linux.

To build aarch64-linux binaries we can:

Build natively on an aarch64-linux machine.
Cross compile for aarch64-linux.
Compile with an emulator.

The first option is the simplest. For example, a Raspberry Pi with Nix installed compiles the aarch64-linux binaries reasonably well. Typically, you need to build only a few packages from source and the rest may be downloaded from the Nix public binary cache. You can also use a more powerful ARM 64 computer for building, if you own or rent one.

I have little to say about the second option because I never managed to get it working. I also suspect that cross compiled packages are not in the public binary cache, so build times are longer than native builds.

In the rest of the article I explore the third option: compiling with an emulator, in particular with QEMU.

On NixOS

On NixOS, it’s trivial to use QEMU to build for different architectures. You need to adjust one parameter in your NixOS host configuration:

boot.binfmt.emulatedSystems = [ "aarch64-linux" ];

This snippet installs and configures QEMU and enables emulated builds for the Raspberry Pi. For example, to build hello from source, run:

nix build nixpkgs#legacyPackages.aarch64-linux.hello --no-substitute

For this demonstration I added the --no-substitute flag to disallow binary substitutes. In other words, this flag forces Nix to build all packages from source.

If the compilation with the emulated tool chain works, Nix writes an ARM 64-bit binary at ./result/bin/hello

Using GitHub Actions

Recently I discovered the setup-qemu-action from Docker which helps us to configure a hosted GitHub Action runner to build for the Raspberry Pi:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-qemu-action@v3                                     # ⑴
      - uses: cachix/install-nix-action@v23                                   # ⑵
        with:
          extra_nix_config: "extra-platforms = aarch64-linux"
      -run: |                                                                 # ⑶
        nix build nixpkgs#legacyPackages.aarch64-linux.hello --no-substitute

This GitHub Actions workflow starts an Ubuntu virtual machine and installs Nix:

Install the QEMU static binaries using a GitHub Action from Docker.
Install Nix using a GitHub Action from Cachix and configure it to allow building for aarch64-linux.
Build hello for aarch64-linux.

Summary

Using two GitHub Actions from Docker and Cachix I set up a workflow to build packages for the Raspberry Pi using freely available Ubuntu runners from GitHub.

In a related article I explain how I use this technique to build and deploy NixOS on Raspberry Pi.

Managing the development of large software systems

2023-02-28T00:00:00+01:00

This is a review of the paper Managing the development of large systems by Winston W. Royce, originally published in August 1970.

Figure 2 of this paper often cited as the “waterfall” method, a model that considers the project’s activities as a linear sequence of steps. In fact, this figure is only the starting point of the paper. Royce acknowledges that the depicted model doesn’t work in practice and he suggests five concrete steps that are required for large software projects to succeed.

Analysis and coding

The paper starts with the observation that there are two essential steps common to all computer program development: analysis and coding.

The author doesn’t discuss these steps in detail. I understand the first step as analysis of the problem at hand and the second as the activity spent typing in the code, debugging, and deploying. In other words, first we define what the desired program is indented to do, then we implement it.

Royce claims that this simple, two-step development model works for systems where the final product is operated by the those who built it. The rest of the paper extends this model to explain the development of larger systems:

”[…] to manufacture larger software systems […] additional development steps are required, none contribute as directly to the final product as analysis and coding, and all drive up the development costs.”

We will see later that by additional development steps the author means program design, defining requirements, writing documentation and testing. It seems that in already in 1970 many people didn’t understand that a software project is much more than just coding:

“Customer personnel typically would rather not pay for them [the additional steps], and development personnel would rather not implement them. The prime function of management is to sell these concepts to both groups.”

Nevertheless, Royce holds the management responsible to explain the structure of the project and the purpose of each step in it.

The author managed the “development of software packages for spacecraft mission planning, commanding and post-flight analysis”. Drawn from the author’s experience, the paper proposes a development model that is required, but not sufficient, for such software project to succeed.

In the sixties the most expensive and most complex software projects solved scientific and engineering problems, often using hardware with constrained resources. Royce clearly makes his claims in this specific context, so perhaps it’s foolish to extrapolate to the development of any software project.

Not waterfall

The third paragraph introduces the “additional steps” the author hinted in the introduction:

“A more grandiose approach to software development is illustrated in Figure 2. The analysis and coding steps are still in the picture, but they are preceded by two levels of requirements analysis, are separated by a program design step, and followed by a testing step.”

Looking from the manager’s perspective, Royce continues:

“These additions are treated separately from analysis and coding because they are distinctly different in the way they are executed. They must be planned and staffed differently for best utilization of program resources.”

The nature of the work in these steps is different, so it makes sense to think about them separately. Royce doesn’t specify how these steps are executed. In fact, he doesn’t discuss this figure any further because this is just an intermediate step, and he’s not finished presenting his model yet.

Local iterations are insufficient

In Figure 3, Royce shows an iterative relationship between successive development steps.

Royce believes in this approach. He thinks these small, local iterations have value because:

”[…] as each step progresses and the design is further detailed […] The virtue of all of this is that as the design proceeds the change process is scoped down to manageable limits.”

Adapting the design in small chunks has benefits, but he immediately points out that following this process is “risky and invites failure”. The problem is that iterations only occur between preceding and succeeding steps but rarely in more remote steps in the sequence. Royce gives an example:

“The testing phase which occurs at the end of the development cycle is the first event for which timing, storage, input/output transfers, etc., are experienced as distinguished from analyzed. […] if these phenomena fail to satisfy the various external constraints, then invariably a major redesign is required.”

With local iterations only, fundamental design problems surface too late in the project. In effect, the development costs increase and the project runs late.

Before discussing the mitigations, Royce closes this section with a note on the role of coding:

“One might note that there has been a skipping-over of the analysis and code phases. One cannot, of course, produce software without these steps, but generally these phases are managed with relative ease and have little impact on requirements, design, and testing.”

It is not the implementation, but discovering the requirements, finding a solid design and defining a test plan are the risky parts of the process.

Eliminate development risks

In the main part of the paper Royce presents five steps that, in his experience, increase the likelihood of success. These are:

Program design comes first
Document the design
Do it twice
Plan, control and monitor testing
Involve the customer

The process is shown in Figure 4, let’s see each steps in detail.

Program design comes first

In center of Figure 4 is “Program design”, this is where the process starts. Program designers lay out the operational design of the program, they design, define and allocate data processing modes even at the risk of being wrong.

Eventually, all software is about data transformation therefore understanding the data is key to understanding the problem. The recommendation here is detailed and prescriptive:

“Allocate processing, functions, design the database, define database processing, allocate execution time, define interfaces and processing modes with the operating system, describe input and output processing, and define preliminary operating procedures.”

The outcome of this phase is an overview document with a goal that:

“Each and every worker must have an elemental understanding of the system. At least one person must have a deep understanding of the system which comes partially from having had to write an overview document.”

Interestingly, in his 1980 Turing-award lecture, Tony Hoare recounts a story of a cancelled project from the sixties where he felt responsible for the failure because he let the programmers do things which himself didn’t understand.

Document the design

Royce starts the longest section of the paper with a categorical statement:

“The first rule of managing software development is ruthless enforcement of documentation requirements. […] Management of software is simply impossible without a high degree of documentation.”

A verbal record is intangible to be a basis for a communication interface or management decision. Without a written record one cannot evaluate the project’s progress nor its completion.

Documentation serves as the primary means of communication between people on different parts of the project: developers, testers, people in operations. Additionally, good documentation permits effective redesign, updating and retrofitting without throwing away the entire existing framework of operating software.

How much documentation is enough? According to Royce, in 5 million dollar (approximately 38 million in 2023) software project, an adequate specification document would be around 1500 pages long.

Do it twice

If the product is totally original, Royce recommends building a prototype which he calls an early simulation of the final product:

“If the computer program in question is being developed for the first time, arrange matters so that the version finally delivered to the customer for operational deployment is actually the second version insofar as critical design/operations areas are concerned.”

The goal of the prototype is to test which hypotheses hold and identify areas of the design which are too optimistic. Royce highlights that the personnel involved in the pilot must have an intuitive feel for analysis, coding and program design. In other words, the most experienced engineers in the team should be assigned to this work.

Plan, control and monitor testing

This part on testing is the second-longest section of the paper, here I highlight only a few points.

If the project followed the previous steps, most problems should be already solved:

“The previous three recommendations to design the program before beginning analysis and coding, to document it completely, and to build a pilot model are all aimed at uncovering and solving problems before entering the test phase.”

In this section Royce observes that most errors are obvious, and he argues for performing code reviews: every bit of code should be subjected to a simple visual scan by a second party who did not do the original analysis or code.

He also promotes good test coverage: every logic path in the computer program at least once with some kind of numerical check. He puts the bar high: as a customer he would insist on nearly 100% coverage.

Involve the customer

Royce recognizes that even after an agreement, what the desired software is going to do is a subject of wide interpretation. He suggests that the customer is formally involved during design reviews and during the final software acceptance review.

Discussion

I am surprised how many ideas, that we consider as “modern”, are present in this paper from 1970. Designing data flows, pilot projects, code reviews, customer involvement are all mentioned here by someone who managed projects during the sixties. Computing was totally different then, but software projects suffered the same problem as today.

More than fifty years later, I feel that the Royce’s wisdom remains relevant. Many suggestions in this paper were rediscovered or reformulated later, perhaps some are still ignored.

This is a fantastic paper. The original is only 11 pages, I highly recommend reading it.

Acknowledgements

The figures in this article are from the original paper.

Year 2022 in review

2022-12-31T00:00:00+01:00

This year I wrote only three articles (including this one), so this will be a short review of the articles in 2022.

The puzzle of Day 8 in 2021 Advent of Code inspired me to explore logic programming, a lesser known, perhaps forgotten, yet powerful programming paradigm. In this article I explain my first logic program and I discuss the advantages and drawbacks I see with the implementation. I want to learn more about this paradigm because, as Scott Wlaschin explains in this presentation, logic programming systems and libraries may yield elegant declarative solutions to some classes of problems.

In Summer, I started to volunteer at Powerhouse, a local organization helping people to learn programming. I held only a few coaching sessions, but the experience rekindled my interest in teaching. I keep working with them in next year too.

Finally, yet again, in December I spent my free time solving the Advent of Code puzzles.

Thanks for reading and happy 2023!

Advent of Code 2022

2022-12-25T00:00:00+01:00

Since 2020, every year I solve the Advent of Code puzzles. This post, like those of previous years, is an experience report of my 25+ day journey in December.

Spoiler alert: If you’re still working or planning to work on the puzzles stop reading now.

Puzzles

Between December 1 and December 25, a programming puzzle appears every day at Advent of Code website. Each problem has two parts, the second part unlocks after you complete the first. To understand each day’s puzzle, go to the website and read the full description (for example Day 1: Calorie Counting).

Setup

In previous years, I used the Advent of Code to learn a new programming language. I used Rust in 2020, and Clojure in 2021.

This year I stayed with Clojure. I feel it’s a good fit for the Advent of Code. Clojure’s support for interactive development, the built-in data structures, and the core library functions allowed me to write clear, concise code that is almost exclusively about the problem at hand.

I wanted to write idiomatic, readable code. I don’t compete on the global leaderboard, I don’t care about coding speed, code size and performance.

Highlights

This year I enjoyed all puzzles, even the hardest ones. The first two weeks went well and I followed a regular daily schedule: I solved the day’s problem before going to work, sometimes I coded a bit in the evening to polish my solution.

The difficulties started on Day 15. I didn’t pay attention to the problem description and I came up with an overly complicated, long, and buggy solution. Looking back, I should have stepped away from the problem for a day or two. Instead, I kept going and my performance in the next days suffered.

Exhausted from the previous day, I took another hit on Day 16 which was about opening valves to release pressure from a cave. I tried to formulate the solution as a shortest-path problem. My program solved correctly the given example but it ran out of memory when I gave it the real input. I also failed to solve the day as a search problem, as there were too many possibilities to explore. Finally, I looked at solutions online and an implementation using dynamic programming gave me a good solution.

Day 17 was about playing a variant of Tetris. The second part asked how many rows the board has after dropping a trillion rocks. Here I also used some help from others. The solution runs fast, but the implementation is clunky: I made it work but I was too tired to clean it up.

Day 19 was about building mining robots that can mine different resources. From the description, the problem looked similar to that of Day 16. Now I suspect that this similarity was a trap because an efficient solution ended up looking different. I borrowed the key idea from Johnathan Paulson: a depth-first search with a limit on the number of mining robots and the accumulated stock gave a correct and fast solution.

My favorite day was Day 22 which was about moving around on a map with “interesting” boundary conditions. I loved the plot twist in the second part which I don’t want to spoil. Many people gave up here, so I’m proud I made it. Later I’d like to generalize my solution because now it only works on the input generated for my account.

Summary

I had a great 2022 Advent of Code, this is my best year so far. My solutions are decent and they are my own except parts of Day 16, Day 17 and Day 19 where I used external help. Maybe next year I’ll recognize similar problems to solve all days on my own.

Solving the puzzles for 25 days next to my full-time job is exhausting. Perhaps for a few days, I pushed too hard and I should take more breaks next year.

As usual, after solving the puzzles I like to see how others approached the same problem. Here are some repositories I regularly checked:

My solutions are available on GitHub.

Acknowledgments

Thanks Eric Wastl for creating and running Advent of Code.

I enjoyed exchanging ideas and analyzing solutions with my friend Kornél.

Exploring Logic Programming

2022-01-23T00:00:00+01:00

In this article I revisit the problem Seven Segment Search of Day 8 in the Advent of Code 2021 puzzle series. I implement a declarative solution in Clojure using the logic programming library core.logic.

Puzzle

The problem of Day 8 was about decoding the digits of a broken seven-segment display. We can measure the signals on the wires powering the display’s segments, but we don’t know which wire is connected to which segment. The puzzle input is a sequence of activation signals of the display as it renders the digits from 0 to 9 in some random order:

"be" "cfbegad" "cbdgef" "fgaecd" "cgeb" "fdcge" "agebfd"
"fecdb" "fabcd" "edb"

The goal of the puzzle is to decode these signal patterns to understand which digit the display renders.

For example, the first signal pattern "be" must be the digit 1 because this is the only digit that is rendered using two segments. But we don’t know which segment the signals b and e are connected. By looking at the digit 1 there are two possibilities:

 ....       ....
.    b     .    e
.    b     .    e
 ....       ....
.    e     .    b
.    e     .    b
 ....       ....

In fact, the digits 1, 4, 7 and 8 each use a unique number of segments: 2, 4, 3 and 7, respectively, so these are easy to identify. To find the other digits we need to collect more information.

For example, the digits 2, 3 and 5 are each contain five active segments. From this group we can find the digit 3 by exploiting that the rendering of the digit 3 doesn’t change with the digit 1 superimposed. In other words, the activation signals of the digit 3 and 1 only differ in one segment.

Imperative solution

Exploiting the similarities between the digit representations you can come up with your own algorithm to unambiguously identify all digits. My implementation in Clojure looks like this:

(defn decode [patterns]
  (let [length  (group-by count patterns)
        ; Unambiguous digits
        [one]   (get length 2)
        [four]  (get length 4)
        [seven] (get length 3)
        [eight] (get length 7)

        ; Candiates for 2,3,5 and 0,6,9
        c235    (get length 5)
        c069    (get length 6)

        ; Find the rest using similar digits
        three   (find-with-mask one c235)
        nine    (find-with-mask three c069)
        zero    (find-with-mask seven (remove #{nine} c069))
        five    (find-with-mask nine nine (remove #{three} c235))

        [six]   (remove #{zero nine} c069)
        [two]   (remove #{three five} c235)]

    {zero 0 one 1 two 2 three 3 four 4
     five 5 six 6 seven 7 eight 8 nine 9}))

The function decode takes the signal patterns as input and returns a map from signal pattern to digit value. The function find-with-mask implements the insight about the similarity of the digits. You can read the entire code of my solution here.

Clojure may not be your language of choice, but in any other mainstream language the solution would be similar in spirit: a few function calls, set and sequence operations. Each operation is trivial but the decode function is hard to understand even for a person familiar with the problem statement. This implementation is imperative, each line prescribing what to do and the order of operations is critical.

In the next section I show a declarative solution where we only state the problem and let the computer figure out how to solve it.

Declarative solution

In this section we reimplement the decode function using the logic programming library core.logic. A logic program is an expression of constraints upon a set of logic variables. We hand this expression to a solver, in our case core.logic, which returns all the possible values of the logic variables that satisfy the constraints. The core.logic Primer explains in detail how the solver works.

Let’s see the declarative version of the pattern decoder in Clojure core.logic:

(defn decode-logic [patterns]
  (let [length (group-by count patterns)]
    (first (run 1 [q] ; ❶
      (fresh [zero one two three four five six seven eight nine]
        ; ❷
        (== q {zero 0 one 1 two 2 three 3 four 4
               five 5 six 6 seven 7 eight 8 nine 9})
        ; ❸
        (== [one]   (length 2))
        (== [four]  (length 4))
        (== [seven] (length 3))
        (== [eight] (length 7))
        ; ❹
        (permuteo (length 5) [two three five])
        (permuteo (length 6) [zero six nine])
        ; ❺
        (overlay one three three)
        (overlay one nine nine)
        (overlay one zero zero)
        (overlay one five nine)
        ; ❻
        (permuteo patterns [zero one two three four
                            five six seven eight nine]))))))

After a standard function definition and a let-binding the body of decode-logic is a set of constraints derived from our original problem statement:

Run the solver and ask for exactly solution. Our problem has a unique solution, but in other applications you may be interested in many solutions that satisfy the constraints. q, the second argument of run, refers to the final solution.
Search the solution q in the shape of a map from pattern to digit value. This is the same structure we used in the previous section.
Identify the “easy” digits, those with unambiguous number of active segments.
The 5 and 6 character long signal patterns are the permutations of the sequences [2, 3, 5] and [0, 6, 9], respectively.
Overlay rules: for example the digit 1 and 3 yields 3 when on top of each other.
The sequence of activation patterns is a permutation of the digits.

decode-logic encodes only the rules of the puzzle without any details of the program’s execution. In fact, you can rearrange the constraints arbitrarily, the logic expression evaluates to the same result.

The order of the constraints, however, affects the solver’s performance: I found that stating more specific constraints first makes the solver explore the possible states faster.

The beauty of the declarative solution comes with a price, at least in my implementation: decode-logic takes approximately six seconds to run where decode completes within a millisecond. This was my first logic program I’ve ever written in Clojure core.logic so I guess this is fine.

The source code of the implementation is available here.

Summary

I implemented an Advent of Code puzzle as a logic expression in Clojure. A logic program is about the what instead of the how: stating the problem is easier, but reasoning about the program’s performance is harder than compared to an imperative implementation.

You can read about my experience of the 2021 Advent of Code here and this repository contains all my solutions.

Year 2021 in review

2021-12-31T00:00:00+01:00

This year I wrote six posts on this blog (including this one), which is fewer than the ten written in 2020. In the first half of the year I was looking for a new job which sucked up most of my creative energy, but eventually last Summer I changed my employer.

Back in February I noticed how hard it is to know exactly what software versions are installed in a container image and I proposed to build container images using Nix. Nix is still the best tool for packaging and deploying software and I use it everywhere I can.

On a rainy weekend in March I learned how to generate random blog posts with a Markov-chain. This was great fun and in the future I’d like to experiment more with various machine learning algorithms.

In June my mind was on writing. I reviewed two books which helped me to improve my writing and I explained how I practice writing at work and in my free time.

Finally, in December I spent my free time solving the Advent of Code 2021 puzzles to learn the Clojure programming language.

Thanks for being here, let’s see what 2022 brings!

Happy New Year!

Advent of Code 2021

2021-12-25T00:00:00+01:00

I solved the Advent of Code, puzzles. This post, like the one from last year, is an experience report of my 25+ day journey in December.

Spoiler alert: If you’re still working or planning to work on the puzzles stop reading now.

Puzzles

Setup

I decided to learn Clojure and to solve the puzzles in this functional dialect of the Lisp language. In the past few years I believe I watched every available conference talk of Rich Hickey, the creator of Clojure. It was time for a full-immersion into the language he designed.

I knew from last year, when I implemented the solutions in Rust, that learning a new programming language while thinking about challenging puzzles is frustrating. So I decided to learn some Clojure before the puzzle series begins.

In November I read a few chapters of The Joy of Clojure and I rewrote some of my Advent of Code 2020 solutions in Clojure. With this preparation I could write decent code from Day 1 but I often restructured my older programs as I learned more about the language.

Just like last year, I didn’t care about code size, performance optimizations (only when it was the goal of the puzzle) and coding speed.

Highlights

My favorite puzzles were Day 8 and Day 13. The former was about decoding the activation signals of a broken seven-segment display and the latter was about folding a large sheet of transparent paper to reveal a secret code. Both problem statements were funny and they fit well into the Christmas-themed story line.

The core algorithms of Day 12, Day 15 and Day 23 were about graph manipulation. I enjoyed implementing traversals and solutions to the shortest path problem.

The puzzles of Day 9, Day 11, Day 15 and Day 25 were stated on a two-dimensional grid. The task of updating the grid or parts of it appeared in many forms. In some cases I could write expressive code using Clojure’s threading macros and the cond-> macro, but I also feel that some of my grid manipulation code is not idiomatic and hard to understand.

I solved Day 6 and Day 14 using infinite sequences. Here Clojure really shines: the core sequence manipulation algorithms work well together resulting in short and expressive code.

On Day 16, Day 18, Day 21 recognizing trees, recursion and the need for memoization were the key elements to write a clean solution.

I spent the most hours on Day 17, Day 19 and Day 22. After multiple failed attempts to solve these days I looked for hints on Reddit. Many smart people hang out in that forum and there’s a lot to learn from the conversations there.

My least favorite day was Day 24. I didn’t see the point in deciphering someone else’s obfuscated code. I do this enough at work already.

Learning Clojure

Using this puzzle series to learn a new programming language is stressful, but effective. Just like last year, solving these puzzles left me with some thoughts about Clojure:

REPL: One big selling point of Clojure is its support for interactive development. I set up my editor with Conjure and in a few hours I got used to evaluating pieces of code directly from the editor.
Testing: I struggled to set up Cognitect test runner, but once I learned the proper way to structure my project things just worked all right. When I’m using other languages I run the tests from a separate command terminal. With Clojure it’s easy to run the tests without leaving my editor.
Core functions: Clojure core contains a ton of useful functions which are easy to use. I used only one external library: data.priority-map. The built-in documentation is terse and short on examples but ClojueDocs improves this and I used it all the time.
Regular expressions: Last year I spent a lot of time of writing parsers for the input files. This year I used regular expressions almost exclusively and the parsing code is rarely longer than a few lines. Clojure’s literal regular expressions syntax and the function re-seq, which returns a sequence of successive matches, are great.

Proponents of Clojure claim that programs written in this language are short and expressive because they are mainly about the problem at hand and not about classes, accessors, lifetimes and other incidental programming language constructs.

In the past few weeks coding in Clojure I find this claim justified. In the future I’d love to explore how is it to build a large system in the style of programming Clojure encourages, that is using functions and generic data transformations.

Summary

Completing the Advent of Code and learning a new programming language was challenging, but a rewarding experience. After solving the puzzles I like to see how others approached the same problem. Here are some repositories I regularly checked:

The source code of my solutions are also available on GitHub.

Acknowledgment

Thanks Eric Wastl for creating and running Advent of Code.

Thanks Olivier Dormond for exchanging ideas and congratulations for winning our private leaderboard!

Leaving Pix4D

2021-07-15T00:00:00+02:00

After four years at Pix4D, today is my last working day. This article presents my main projects and some lessons learned at this company.

Calibration Team

I joined Pix4D’s Calibration Team in August 2017 as a C++ developer. My first task was to improve the quality report generated by the first stage of PIX4Dmapper‘s image processing pipeline.

Best times: I learned a about photogrammetry.
Worst times: I rearranged a data structure in some reporting code and the performance of a seemingly unrelated module dropped 10%. The team spent weeks tracking down why this happened.
Lesson learned: Fighting code complexity is the hardest problem in maintaining a successful software product.

Integration Team

In 2018 I joined the Integration Team to work on a new build automation system for the company. Developers needed a tool to build and test their applications and libraries on various platforms. I developed an interest in building cloud infrastructure and deployment pipelines.

Best times: I learned about Amazon Web Services and about modern tools for deploying distributed systems.
Worst times: Because of an unpinned dependency we picked up a kernel update with a bug that crippled our entire fleet of build machines.
Lesson learned: When working on a green-field project our head is full of creative ideas and it’s easy to miss prior work.

The experience in this team inspired me to experiment with my own build automation system.

Cloud Services Team

At the end of 2019 I was invited to join Cloud Services, the team responsible for Pix4D’s photogrammetry processing service. I was excited to work on a live system serving paying customers. I also had a chance to lead the technical work of a small team.

Best times: I learned about the operation and maintenance of a production cloud system.
Worst times: Browsing through a hundred SQS queues trying to reverse-engineer what they’re for.
Lesson learned: Building things incrementally is a good idea. Figuring out what not to build is a better idea.

The work in this team rekindled my interest in Nix and led me to build a decent Homelab.

Summary

After four great years at Pix4D I’m moving on. I accepted a job at Nexthink where I keep learning about the intricacies of cloud systems.

I wish everybody at Pix4D a lot of success!

Practicing writing

2021-06-05T00:00:00+02:00

In the past years I have been learning about technical writing. In this article I review two books that helped me to improve my writing skills. Then, I present how I practice writing at home and at work.

Discovering writing

As a scientist I published technical reports, journal papers and even a 200 pages long doctoral thesis, but I knew little about writing, a communication skill that can be learned and improved. I left academia and I got a job as a software engineer. I stopped following science and started learning programming languages and study industry best practices.

One summer I read REWORK, a book written by Basecamp’s founders. The chapter Hire great writers explains the authors’ preference in hiring candidates who can demonstrate great writing skills because clear and efficient written communication is essential in their distributed and asynchronous working style. I was an employee and I didn’t plan to hire anybody, but the emphasis on high-quality writing inspired me to become a better writer.

I looked at my own working habits and, somewhat naively, realized the obvious: I write all the time. Instant messaging, code commits, pull-requests, code reviews, project proposals, incident reports are all a form of writing. I wanted to learn what good writing is and how to write well.

I searched for resources targeted at practitioners who write technical text. In the next sections I present two books from which I learned the most.

Book: BUGS In Writing

The book BUGS In Writing by Lyn Dupré is a catalog of common mistakes. It comprises short, easily digestible segments. Each segment presents a principle to make your technical writing clear and lucid. The principles are illustrated by witty example phrases and numerous cat photos.

I encountered this book through a reference in Chris Okasaki’s thesis Purely Functional Data Structures:

I was extremely fortunate to have had excellent English teachers in high school. Lori Huenink deserves special recognition; her writing and public speaking classes are undoubtedly the most valuable classes I have ever taken, in any subject. In the same vein, I was lucky enough to read my wife’s copy of Lyn Dupré’s BUGS in Writing just as I was starting my thesis. If your career involves writing in any form, you owe it to yourself to buy a copy of this book.

When I read this recommendation I immediately ordered a copy. BUGS in Writing is a fantastic handbook. When I have doubts about a specific word or expression I search for advise in the book.

To give a taste of Dupré’s style, here’s an example from the segment Impact, where you learn about the one of the most overused, ugly words in the English language:

There are only two pleasing uses of impact: to denote a forceful collision, and to mean packed or wedged in.

Here the principle is that:

[…] you should not use impact when you mean influence or effect, and you certainly should not use impact when you mean affect, because impacting people is incredibly impolite.

When you learn about such principles you develop ear for detecting bugs in texts and your tolerance for buzzwords and corporate jargon decreases.

Book: On Writing Well

William Zinsser On Writing Well is really what its subtitle claims: the classic guide to writing nonfiction. This book showed me how a professional writer thinks about writing methods, style, structure and genres. The book has a chapter on writing about science and technology, but it also treats business writing, writing about arts, sports and even memoirs.

Zinsser warns about clutter, every piece of text that doesn’t do useful work. He suggests stripping every sentence to its barest bones: delete pompous words, redundant adjectives, jargon and replace laborious phrases with short words that mean the same thing.

Writing a clean sentence is hard work. And the work is non-linear and iterative: rewriting is the essence of writing as Zinsser puts it.

Writing a blog

In 2013 I published the first article on this blog. I wanted to write, but I didn’t know what to write about. A common dilemma of the aspiring writer. In a few years I published a handful of short articles on random topics. Some of them were meta-articles on setting up this blog. Some were as simple as connecting a LED and a button to a Raspberry Pi. Unexpectedly, I had a few “hits” too: many friends liked the article on my working habits.

Around 2017 I changed my writing strategy. Instead of overthinking the subject of my next article, I decided that my hobby projects should end with an entry on my blog. While working on a project I take notes and write outlines. When I’m finished (or bored) I have a seed of a new article. I still have to work hard to make my notes into a coherent story, but I feel that the goal, the published article, is reachable.

Writing at work

In my team we improve code using pull-requests, changes that typically comprise few dozen to few hundred lines of code. We prefer small batches of code changes that are easy to review. The risk of breaking existing systems is lower than if we changed everything at once. But through these incremental changes it’s hard to see the context in which a given pull-request makes sense.

I felt the team needed a document that shows how each code change fits into the story of the project. Earlier, my team had considered writing change request documents but some felt it bureaucratic and, frankly, we didn’t know how to write them well.

Last December at work I started a new project that required changes in many subsystems. I described my implementation plan on the company wiki. In few sentences I created a design document.

I started the implementation and as I learned more about the problem domain I refined the design and I updated the document with my progress. If you’d read the document you would have known what’s been done and what’s left to do. The wiki page became a project page.

Of course my initial design had holes and I discovered traps and dead ends. I recorded my failings on the same wiki page which evolved into an experience report.

When I finished the implementation I collected instructions for operators, the support team and for other developers, transforming the page into project documentation.

A few months later when I reread this page I found a historical record that explains why and how we arrived to state we have today.

During this process I found a few practices useful:

I rewrite the same document as we work on the project. The document is always the entry point for all project-relevant information.
I write prose with complete sentences to build context. A few words with bullet points, list of ticket IDs are insufficient.
I often reference related content. There’s no need to repeat what’s explained elsewhere.
I share every major revision with my colleagues asking for reviews and suggestions.

The time spent evolving this page was worth it. I felt that the state of the page reflected the stage of the project. If a project needs long pages to explain, it’s too big. If a few words can’t express what I’m doing, the project is ill-defined, or I lack some knowledge to execute it.

Summary

I wrote during my whole career, but only lately I have been cultivating my writing skills. I practice writing in my free time and professionally. I document my hobby projects and software experiments on my blog and I write “evolving” documents on the company’s internal wiki.

Markov-chain word generation

2021-03-07T00:00:00+01:00

Recently I reread some chapters of the book The Practice of Programming by Brian Kernighan and Rob Pike. In Chapter 3 on Design and Implementation the authors present several implementations of a random text generator to compare how various languages’ idioms express the same idea.

I wrote my version in Rust which I present in this article with some example text I generated using various sources.

Random text

The goal is to write a program that generates random English text that reads well. The program takes a large body of text as input to construct a statistical model of the language as used in that text. Then the program generates random text that has similar statistics to the original.

Here’s an example generated sentence, given the words of all articles of this blog:

In this article resonate with you I recommend to watch the original talk on YouTube with meticulously prepared slides, what we see similar configuration blocks everywhere.

This is obviously nonsense, but the program may generate some funny phrases, especially if the input is long and varied.

The program uses a Markov-chain algorithm to generate text. The words of the input text are arranged in a data structure that records which two-word combinations are followed by which words.

For example, let’s assume that in the input text the words In this are followed by one of the following words: example, post, workshop, specific, representation. Starting from In this, we generate the next word by choosing randomly from these candidates. Let’s say we pick specific. Then, in the next iteration we repeat the same procedure for the words this specific. We can continue generating new words until we have candidate words to pick from.

Rust implementation

I wrote my version of the word generator program in Rust. This code snippet shows its basic usage:

let mut builder = ChainBuilder::new(2);         // ①
for word in words {
    builder.add(&word);                         // ②
}
let chain = builder.build();                    // ③
println!("{}", chain.generate(100).join(" "));  // ④

Create a ChainBuilder. The chain will use two-word prefixes to detemine the next word.
Add words from the input text and store them in a hash-map internally.
Finish building and create the actual Chain.
Use the Chain to generate 100 words and print them as a sentence.

The whole implementation is 129 lines long and it’s available on GitHub. It’s a complete command-line application with help texts and proper argument parsing. The used algorithm and data structures are identical to those described in the book. You can also find the C, C++, Java, Perl and AWK versions written by Kernighan and Pike here.

Examples

Let’s see some generated texts based on different data sets. The prefix length is always two. The code gives a random text at each execution and I ran the program a few times until I obtained a variant which I found funny or interesting. Often I stripped the unfinished, partial sentences from the end of the text.

Book of Psalms

Kernighan and Pike use the Book of Psalms from the King James Bible as a test dataset because it has many repeated phrases (Blessed is the…) which provides chains with large suffix sets. I started to test my program with this data set. Here’s one example:

Blessed is the lord. Praise the lord, at the blast of the enemy: thou hast maintained my right hand is full of water: thou preparest them corn, when thou hast known my soul had almost dwelt in silence.

Indeed, the characteristics of the input dataset are recognizable in each random output, the text reads like prayer using archaic English words.

Technobabble

Richard Feynman’s observations on the reliability of the Shuttle is a fantastic read on it own. The 5000 word long eloquent, technical text is an excellent source for technobabble generation.

It appears that there are three engines, but some accidents would possibly be contained, and only three in the second 125,000 seconds. Naturally, one can never be sure that all is safe by inspecting all blades for cracks. If they are found, replace them, and if none are found in the list above). These we discuss below. (Engineers at Rocketdyne, were made.

In this example we can see that the implementation has no notion of punctuation: unmatched parentheses appear at random places because they are considered to be part of the word.

Here’s another one:

It appears that there are enormous differences of opinion as to the Space Shuttle Main Engine. Cracks were found after 4,200 seconds, although usually these longer runs showed cracks. To follow this story further we shall have to realize that the rule seems to have been solved.

Blog corpus

Finally I tried to generate a random blog post using the articles I’ve written here.

I used Pandoc to convert the markdown files I’ve written to plain text:

pandoc  *.markdown --from markdown --to plain

This yields about 20 thousand words of text, some of which appear in equations, enumerations and code blocks. I wrote a function to ignore almost everything but the actual text. Here are two examples which show that the program is not quite ready to replace me in writing new articles:

In this representation we can assume that we exploit that IO is a simple function. Its imaginary type signature is: This reads: the configuration blocks we implicitly created a simple function. Its imaginary type signature is: This reads: the configuration code on GitHub.

In this niche domain the utility of functions is well studied and understood. Ideas from purely functional programming language using YAML’s syntax. In programming we use fmap to lift our pure project function to be Python, but it was served from. We now are ready to define components, component configuration and component relationships.

In these outputs I can recognize full sentences of my previous articles which shows that the blog corpus is not large enough for automatic blogging.

Summary

Using a Markov-chain to generate random text in a given “style” is a fun programming exercise. The problem can be solved in a few dozen lines of code, but it teaches a lot about algorithms and design in general.

Building container images with Nix

2021-02-25T00:00:00+01:00

Dockerfiles are de facto standard for creating container images. In this article I highlight some issues with this approach and I propose building container images with Nix.

Container images

Many compute engines adopted container images as an interface between your code and their runtime platform. You package your application in a self-contained bundle and hand it over to your compute engine of choice for execution. The compute engine’s runtime system creates one or many container instances based on the provided image and runs your application using some form of isolation.

Typically the application doesn’t have access to any software packages installed on the physical computer where it runs. Therefore, the application can only run correctly in a container if all of its software dependencies are also packaged in the same image.

Dockerfile

The canonical way of creating container images is via a Dockerfile. For example, let’s create an image for a web application written in Python:

FROM alpine:3.5  # ①

RUN apk add --update py2-pip  # ②

RUN pip install --upgrade pip  # ③

COPY requirements.txt /usr/src/app/  # ④
RUN pip install --no-cache-dir -r \
    /usr/src/app/requirements.txt  # ⑤
COPY app.py /usr/src/app/  # ⑥
COPY templates/index.html /usr/src/app/templates/  # ⑦

EXPOSE 5000
CMD ["python", "/usr/src/app/app.py"]

This is a short, working example which you can use to deploy a non-trivial web application. But if you look closely, there are some issues with this Dockerfile. Let’s see them, line by line:

The FROM keyword indicates that we’re starting from a base image. In other words, we define a dependency on a binary blob. You may not know what programs and libraries the selected base image contains. In this specific case, the Alpine Linux Docker images are described in this repository.
The apk package manager, part of the base image, installs some version of the Python interpreter which is required to run our application. Because we don’t specify the exact version, every time you build this image you may install different versions.
We install pip which we’ll need at subsequent build steps, but not when the application is running. The problem of the unpredictable versions, mentioned in the previous item, applies here as well.
We copy the requirements.txt to the image. This also only required for building the image.
The pip package manager installs the run-time dependencies of our application. If requirements.txt specifies exact version numbers we know exactly which packages will be in the final image.
We copy the application, which is only a single file in this case.
We copy an HTML template, also required during run-time.

Each line transfers some component into the final container image. However, the Dockerfile‘s imperative language fails to capture precisely our application’s dependencies:

Extra components: some dependencies are only needed to build the image, some are essential for the application to execute correctly. Yet, both kinds end up in the final image. For example: pip, apk, and requirements.txt are not needed in the final image.
Implicit dependencies: The image contains all packages from the base image but it’s not clear which ones are actual dependencies.
Non-determinism: The version of most of the dependencies are not pinned. At each build different versions may be installed.
Package managers: we use four different ways to manage dependencies: base image with the FROM keyword, pip, apk and copying files.

We find Dockerfiles with such structure used by many software projects. I took this section’s example from a Docker for Beginners repository. In the next section we’ll see how novice users write their Dockerfiles.

Dockerfile from scratch

The ideal container image contains the application and the application’s dependencies and nothing else.

Kelsey Hightower shows how to build such a container image for an application called weather-data-collector using the following Dockerfile:

FROM scratch
COPY weather-data-collector .
ENTRYPOINT ["/weather-data-collector"]

This copies the application’s locally built executable on the image’s file system. The image contains no shell, no package manager, no additional libraries. Both image size and its the attack surface is minimal. This technique does work and creates an ideal container image and does solve all the issues raised in the previous section.

There’s a limitation though: this Dockerfile assumes that the application is built as a single, statically linked binary.

Also, by reading this Dockerfile we don’t know anything about provenance of the application’s binary: which build commands were executed, which tool chain version is used, what additional libraries were installed before building. To remedy this it’s common to use multi-stage Dockerfiles. The first stage prepares the build environment and builds the application’s binary. Then, during second stage, the binary is copied into an independent layer.

Multi-stage builds may work well for you if you always run the application in containers, even during development. In my experience, however, the build instructions in the Dockerfile quickly become the duplicate of the project’s native build instructions.

Blame your build system

I don’t believe that writing a Dockerfile is inherently bad. I think the underlying problem is that our build systems do a poor job of capturing our application’s dependencies. The various Dockerfile tricks are just workarounds for the absence of proper dependency management.

Take a look at any non-trivial project’s build instructions. I bet you’ll find a getting started guide with a long list of tools and libraries to install before you can invoke the project’s build system.

This is admitting that the project’s dependency lists are incomplete and you need out-of-band manipulations such as installing a package or downloading something else from the Internet.

Container images with Nix

Nix was explicitly designed to capture all dependencies of a software component. This makes it a great fit for building ideal container images which contain the application, the application’s dependencies and nothing else.

To demonstrate, here’s an example which packages the HTML pages of this blog and a web server into a container image.

containerImage = pkgs.dockerTools.buildLayeredImage
  {
    name = "thewagner.net";
    contents = [ pkgs.python3 htmlPages ];
    config = {
      Cmd = [
        "${pkgs.python3}/bin/python" "-m" "http.server" 8000
        "--directory" "${htmlPages}"
      ];
      ExposedPorts = {
        "8000/tcp" = { };
      };
    };
  };

You can run the built image yourself:

docker run -p 8000:8000 wagdav/thewagner.net

If you visit http://localhost:8000 you can read my articles served from the locally running container.

This snippet calls the function buildLayeredImage from the Nix Packages Collection to build the container image. The function’s arguments define the image’s name, contents and its configuration according to the OCI specification. The function returns a derivation which builds a container image containing the Python interpreter, for the web server, and the static pages of my blog.

In short, you list the components you want in the image and Nix copies those and their dependencies into the image archive. Perhaps this summary sounds underwhelming, but proper dependency management renders the building of the container images conceptually simpler. Instructions in a Dockerfile are analogous to a shell script provisioning a freshly installed computer. The Nix expression looks like creating a compressed archive specifying a list of files.

To use buildLayeredimage you must have a Nix expression to build your application. The Nixpkgs manual has a long section on building packages for various programming languages.

Concretely, for the example showed in this section, you can find the complete definition of htmlPages here. Everything is built from source and the pkgs attribute set points to a specific commit of the Nix Packages collection. The image is reproducible: no software version changes between builds unless explicitly updated.

As a bonus, this image is automatically built by GitHub Actions every time I push new content in the source repository. And this works without fiddling with QEMU, BuildX and Docker or using a special remote Docker setup.

Summary

Dockerfiles, when used naively, yield bloated container images with non-deterministic content. The various techniques such as building static executables and using multi-staged builds are just workarounds for the absence of a programming language independent build system which is capable of capturing all dependencies of an application.

Using Nix and the Nix Packages Collection it’s possible to build minimal, reproducible container images with a few lines of code which run locally and on any hosted build automation system.

Year 2020 in review

2020-12-31T00:00:00+01:00

This blog has no specific theme, I write about topics that are on my mind at the moment. In this post I reconstruct a story arc of the year and link to articles I wrote in 2020.

During the holiday break in 2019 I wrote about functions in disguise. I argued that we should use pure functions in software configuration files instead of letting them develop idiosyncratic concepts and custom rules.

I was particularly frustrated by the configuration of build automation systems (also called continuous integration, continuous delivery systems). In January I was researching the essence of a software delivery pipeline and I picked on the concepts of some popular tools. I argued that a software delivery pipeline could be viewed as a function without introducing other, ambiguous concepts. To validate this idea I wrote Kevlar, an experimental build automation system where pipelines are functions. In Kevlar, contrary to many of our tools, parallelism is completely automatic.

While thinking about software delivery pipelines I discovered Nix which now I use to configure all computers in my homelab. Using Nix I made this blog’s deployment automatic and reproducible. I believe Nix is the best tool today for configuring physical machines and for building software delivery pipelines.

In the second part of the year I’ve been learning Rust. I explored its concurrency primitives and in December I completed the Advent of Code in Rust too.

In summary, the articles I wrote over the past year fall in three broad categories:

Concepts and configuration of build automation software
Infrastructure and Nix
Learning the Rust language

Looking forward to seeing what 2021 brings!

Happy New Year!

Advent of Code 2020

2020-12-25T00:00:00+01:00

This year I participated the first time in the Advent of Code, a series of programming puzzles before Christmas. This post is an experience report of my 25+ day journey in December.

Spoiler alert: If you’re still working or planning to work on the puzzles stop reading now.

Puzzles

From December 1 to December 25, a programming puzzle appears each day at Advent of Code website. To understand each day’s two-part puzzle, go to the website and read the full description (for example Day 1: Report Repair).

Setup

I decided to write all solutions in Rust because I wanted to learn the language. This was my only “hard” requirement, the rest was just to have fun. After solving the first couple of puzzles I set the following boundaries for myself:

Language: I write the solutions in Rust with minimal external dependencies. I used only the regex and itertools crates.
Independent programs: Each day’s solution is a separate, independent executable with no code reuse among the solutions.
Tests: I always transform the provided examples into automated tests. Sometimes I add more tests, but I’m not strict about them. GitHub Actions run the tests on every commit.
Explore: I rewrite code as I learn new features of the Rust language and its standard library. The code what you see now in the GitHub repository is often not my first attempt.

I also decided what were not my goals:

No code golf: I wanted readable and concise solutions, but in no way I tried to minimize the lines of source code.
Coding speed: I had no ambition (or chance) competing on the global leaderboard. I solved the problems at my own speed, next to my full-time job. Often it took me days to finish a puzzle.
Code performance: I didn’t measure or optimize my code’s performance in any way. Nevertheless, the execution time of every solution is less than a few seconds.

Highlights

Variations on Conway’s Game of Life appeared on multiple days: on day 11 the cells evolved on a fixed grid, on day 17 in three and four dimensions, and on day 24 the solution was expected on a hexagonal grid. I enjoyed these problems and learned a lot about Rust iterators and iterator adapters.

Day 18 was about a small expression evaluator with custom precedence and associativity rules. I learned about Dijkstra’s shunting yard algorithm and I used it to transform the expressions specified in infix notation to reverse Polish notation.

The second part of the puzzle on day 13 caused me headaches. Someone on a Reddit thread suggested that the Chinese remainder theorem is relevant here. I learned the basics of this theorem and recognized how to use it in the solution.

Day 10 took me multiple days to finish. I failed recognize that relevant strategy to solve this puzzle was dynamic programming on which I even wrote an article. Perhaps I should read it again…

In the weeds

There were also days when the problem itself was not particularly difficult, but I still struggled. On these days I ended up with code that sort of works, but not particularly nice:

Day 19: The problem input represented a language akin to regular expressions. My parsing code became quite messy, maybe I should have used actual regular expressions to parse the input file.
Day 20: I made many mistakes during the implementation, the solution was tedious with much bookkeeping and little reward.
Day 21: I couldn’t connect to the problem because I found it too convoluted. After I looked at other people’s solution it was clear that I was missing the underlying search algorithm therefore my implementation turned out to be less than great.

Learning Rust

Solving these puzzles was great for learning Rust. From these last three weeks I’d underline these topics:

Cargo: The cargo package manager works well and easy to use. The project was easy to start and the file layout is clear and simple.
Testing: Rust has a built-in testing framework that integrates well with cargo. It’s easy to add tests and maintain tests because the test code is close to the application code.
Cargo watch: Cargo Watch can run the tests every time a change occurs in project’s source. I only introduced this tool at day 16. I should have done it right at the beginning.
Vec: For almost every puzzle I used Vec from the standard library.
Iterators: I learned to use methods of the Iterator trait. I used map, filter, count almost every day. Long iterator chains take a bit of getting used to, but they are idiomatic in Rust.
itertools crate: multi_cartesian_product from the itertools crate was a life-saver for implementing Game of Life in arbitrary dimensions.
Regex crate: Well, regular expressions are just everywhere.
enum: In Rust enum is a real sum type where the variants can contain data too. I used an enum as a central type on days 4, 8, 14, 18 and 19.

Summary

Completing the Advent of Code and learning a new programming language was challenging. I learned a lot in these last 25 days about puzzles, algorithms and about the Rust language.

I particularly enjoyed the puzzles where knowing an algorithm is rewarding and makes your code concise and readable. When my code is long and messy I know that I’m missing something: a data structure, an algorithm or a language feature. The hard part is to figure out which one.

The source code of all solutions are available on GitHub.

Acknowledgment

Thanks Eric Wastl for creating and running Advent of Code.

Deploying with GitHub Actions and more Nix

2020-12-06T00:00:00+01:00

In July I described how I use Travis CI to deploy this static site to GitHub Pages using a Nix pipeline. Before continuing I suggest reading that article because rest of this post builds on top of that.

This article is about the changes I made in this blog’s deployment process during the last months. These include switching to Nix Flakes, adding more checks to the pipeline and moving from Travis CI to GitHub Actions.

Flakes

Flakes are an experimental mechanism to package Nix expressions into composable entities. Flakes define a standard structure of Nix projects for hermetic and reproducible evaluation.

The blog is still built using Nix, but now the entry point is flake.nix. The heart of this expression is a function:

outputs = { self, inputs.. }:
{
  checks = ..
  modules = ..
  packages = ..
  ..
}

The function outputs takes the project’s external dependencies as inputs and returns the build artifacts in a record. Build artifacts can be Nix packages, NixOS modules, test results, container images, virtual machine images. Basically anything Nix can build. We’ll see a concrete example of the checks attribute in subsequent section. There’s an ongoing effort to define a standard structure for the returned value so that specific tools can understand and use the built derivations.

The source tree also contains a lock file flake.lock to ensure that pages of this blog are always built with exactly the same set of tools independently where the build is executed. The same environment is used on my workstation, on yours, on a worker of a hosted CI system.

To learn more about Flakes I recommend the following resources:

Presentation by Eelco Dolstra at NixCon 2019 (Youtube)
Three-part series on Tweag’s technical blog (part 1, part 2, part 3)
RFC documenting the flake’s structure (link)

Compatibility

Currently flakes are unstable and experimental in Nix. You need to explicitly enable flake support if you want to use them. The repository flake-compat provides a compatibility function to allow flakes to be used by non-flake-enabled Nix versions.

If you install the Nix package manager on your platform and clone the repository you can build the static pages of this site by running nix-build in the source tree.

With a flake-enabled, experimental Nix version you can even build the project without cloning, directly referencing the repository on GitHub:

nix build github:wagdav/thewagner.net

Building the static pages of this blog has no practical use for anybody but me. But imagine if your favorite project would build on your machine without installing anything but one standalone binary?

Language-specific package managers such as cargo, go get, npm, pip work well if your project uses only that specific language. The reality is that even the simplest projects, such as the source code of this blog, may require tools from any language ecosystems.

Checks

The checks attribute of the structure returned by the flake’s outputs function describes self-tests. For this blog’s source the checks look like this:

outputs = { self, nixpkgs }:
  checks.x86_64-linux = {

    build = self.defaultPackage.x86_64-linux;

    shellcheck = pkgs.runCommand "shellcheck" { .. }

    markdownlint = pkgs.runCommand "mdl" { .. }

    yamllint = pkgs.runCommand "yamllint" { .. }
  };
};

The checks are grouped per supported platform, in this case there’s only one: x86_64-linux. For this blog’s source checking means:

Build the static the blog’s static HTML files
Run shellcheck on all scripts in the source code
Run mdl on all markdown files in the source code
Run yamllint on all YAML files in the source code

If any of these steps fail, the project is considered broken. You can see the full code here.

If you use the experimental Nix version with flake support you can execute all the checks with the following command:

nix flake check  # using Nix experimental

Or with stable Nix without flake-support:

nix-build -a checks.x86_64-linux  # using Nix stable

Again, running the checks only requires the Nix package manager to be installed.

GitHub Actions

Previously the build and deployment scripts ran on Travis CI. I was curious to see how the deployment would work on GitHub Actions, which has become popular during the past year.

The transition from Travis CI to GitHub Actions was trivial. The workflow definition contains the minimal required boilerplate. 21 lines specify the following steps:

Check out the repository
Install Nix using Cachix’s install-nix-action
Run the checks
Deploy the site if on the master branch

The workflow is not concerned with installing or configuring anything but Nix and it’s merely coordinating the build and deploy steps.

Summary

I use a Nix expression to build the static pages of my blog. The build runs locally and on the workers of hosted CI/CD systems such as GitHub Actions and Travis CI. The build requires no external dependencies other than the Nix package manager. The build is reproducible and hermetic: no matter where the project is built, which packages are installed, the result is always the same everywhere.

Kevlar

2020-10-18T00:00:00+02:00

Previously, I distilled the essence of a software delivery pipeline and argued that a transformation step that builds or tests a piece of code can be viewed as a function. Functions compose according to well-defined mathematical rules and they are a suitable model for defining arbitrary pipelines. Instead of talking about build tasks, jobs, stages and workflows, the build pipeline could be a function that takes the source code as argument and returns build artifacts.

In this article I describe, Kevlar, an experimental build automation tool that tries to build on these concepts.

Programming in YAML

For almost two years I was part of a team building a build automation system for Pix4D, a medium-sized organization with:

Few dozen developers
Handful of projects
Couple of programming languages

We wanted a system to build our software products on all the major desktop and mobile platforms. A few years back when the project started the scene of managed build automation tools was less exiting than today: Travis CI was a a major player, GitLab was on the rise, GitHub Actions and CircleCI didn’t exist.

For various reasons we ruled out the available hosted options and we deployed Concourse CI in our data centers. Concourse served us well: today Pix4D’s continuous integration system builds more libraries and products than ever. Concourse uses YAML for describing its pipelines which model the software delivery process. And YAML started to sprout everywhere.

Although YAML is used to configure virtually all build automation systems its limitations become apparent when you write a pipeline for any reasonably complex software project. Using only numbers, strings, lists and associative arrays pipeline definitions are verbose and repetitive.

To allow for reusing pipeline code build automation systems introduced ad-hoc concepts and workarounds. Some examples are:

Many systems also provide control flow operators such as conditionals, loops and other special constructs to express, for example, matrix builds. Ultimately the pipeline configuration becomes an implicitly functional programming language using YAML’s syntax.

In programming we reduce repetition using methods, functions and procedures. We organize our code in modules and packages for sharing and reuse. Let’s try to use these concepts to express software delivery pipelines.

Enter Kevlar

I found programming in YAML frustrating and I wanted to explore how a build pipeline would look like in a general-purpose programming language. Project Kevlar was born.

In Kevlar you express the build pipelines using Haskell functions. A step that builds Kevlar itself looks like this:

build repo = do
  src <- clone repo
  img <- Kaniko.build
    "kevlar-builder"
    "docker/kevlar-builder"
    [Need src ""]
  shell
    ["./ci/build.sh"]
    [ Need src "",
      Image img,
    ]

This function takes the source repository as a parameter and describes the following actions:

Clone the source repository.
Run Kaniko to build a container image described in a Dockerfile of the source repository.
Start a container from the build image and execute the build script.

The Need argument of the functions express data dependencies explicitly. We need the cloned repository to build the Docker image because the Dockerfile is found there. Similarly, because the build script is running in a container it needs both the source code checked out and the container’s image built.

The syntax may be unusual, but it’s just regular Haskell code. This could have been easily written in YAML with equal clarity, so why bother with Haskell? The advantages of using real programming language constructs appear when we build more complicated workflows. As pipelines grow we can factor out common into helper functions or shared modules and packages.

Let’s continue building Kevlar’s own pipeline by adding a publish step:

publish repo binary = do
  src <- clone repo
  img <- Kaniko.build
    "kevlar-publish"
    "docker/kevlar-publish"
    [Need src ""]
  shell
    ["./ci/publish.sh"]
    [ Need binary "build",
      Need src "",
      Image img,
      Secret "GITHUB_ACCESS_TOKEN"
    ]

The publish function takes the address of the source repository and the build binary. The steps are similar to that of the build function:

Clone the source repository.
Build a Docker image with the tools required for releasing the binary artifacts.
Start a container from the built image and execute the publish script.

With the build and publish functions we can succinctly define Kevlar’s build-and-release pipeline:

buildAndPublish src = build src >>= publish src

This function, which composes build and publish using the monadic bind operator, actually works it does what you’d expect: the pipeline builds and publishes the built binaries running both steps in dedicated Docker containers.

We may refer to the functions build and publish as “steps” and the function buildAndPublish as the final “pipeline”. In Kevlar they are all represented as functions returning a Task value.

The function buildAndPublish defines the following graph of dependencies:

           ↗ build image  → compile ↘
clone repo                            publish
           ↘        build image     ↗

This figure reveals interesting optimization opportunities:

We could run the image building steps in parallel because they don’t use each other’s output.
We could avoid cloning the repository twice and reuse the repository’s local copy in downstream steps.

The next sections expand on these ideas in detail.

Automatic parallelism

Instead of naively executing each steps as they appear in the source code, Kevlar uses as much parallelism as possible. The user defines data dependencies using the Need parameter and parallelism is automatic. This idea is not new: for example build systems like make and ninja track dependencies among build steps and schedule as many of them as they can on the available processors.

Initially I built Kevlar on top of Shake, a library for creating build systems. Shake is a well-designed and performant library but it was a poor fit for Kevlar. Shake takes the source code and builds your program’s binary as fast as possible. Build rules and dependencies between tasks rely on file names and file patterns. I needed a library to express general data dependencies in the pipeline code without using the file system.

I switched Kevlar to use Haxl to make parallelism automatic. Haxl relies on some algebraic properties of the program to identify data dependencies between tasks and schedules many independent tasks concurrently. The magic of Haxl is contained within its own codebase and I only had to implement a custom data source. Even better, the user, in this case the pipeline author, doesn’t need to be aware of any of this and the pipeline remains a regular Haskell function.

Today’s popular build automation systems require the user to explicitly choose between sequential or parallel execution when designing the pipeline. Concepts like tasks, steps, jobs are introduced with an emphasis of execution order instead of capturing the meaning of the transformation steps in the software delivery process.

In Kevlar the user is only concerned with data dependencies. The system makes sure that the pipeline’s task run in the right order as fast as possible.

Incremental work

Avoiding extra work is crucial for good performance. The output of a given task should be reused as the pipeline executes: the output might be available from earlier steps or from earlier executions.

Reusing a result is desired if the pipeline fans out: we compute the result once and copy it to the downstream tasks. We’ve seen an example of this in the previous section where the repository’s local copy was passed to start building the two container images independently.

Reusing results from earlier executions is harder. Tasks may return many kinds of outputs: single files, directories, container images. These all need to be persisted somewhere to be reused during the next pipeline execution. I couldn’t find a satisfactory solution to this in Kevlar.

Summary

Frustrated by the verbose YAML configuration used by popular build automation tools I wrote Kevlar, an experimental system, where the pipeline configuration is expressed in a functional programming language.

Pipelines are functions which define data dependencies and not execution order. With some help from great libraries parallelism is automatic and duplicate work is avoided.

Working on Kevlar made me appreciate even more the power of pure functions: values, functions and their combinations are powerful modeling tools. When thinking functionally you ask what things are instead of what they do and this leads to interesting discoveries and simple, solid designs.

I also realized that I’m rediscovering the basic principles of Nix and Nix is so much better than Kevlar ever wanted to be. If the ideas in this article resonate with you I recommend to try building your next software delivery pipeline with Nix.

Acknowledgment

I dedicate this post to the memory of my friend and colleague Salah Missri who tragically passed away earlier this year. Salah was the biggest fan of Kevlar. He patiently listened to me ranting about continuous integration systems and encouraged me keep working on Kevlar until it reaches world domination.

Concurrency in Go, Clojure, Haskell and Rust

2020-07-10T00:00:00+02:00

In the past I wrote two articles where I explored concurrency in Haskell using some examples from the talk Go Concurrency Patterns by Rob Pike.

The examples are different implementations of a simulated search engine which receives a search query and returns web, image and video results. The first version sends the search queries sequentially. Then, the program is gradually improved to become concurrent and better performing.

Earlier I presented each step in detail. Here I will only show the final form of the fake search function in four different programming languages: Go, Clojure, Haskell and Rust.

Go

Rob Pike’s version of the search function executes the three kinds of search queries concurrently and sends the search requests to replicated back-end servers to reduce tail latency.

// https://talks.golang.org/2012/concurrency.slide#50
func search(query string) (results []Result) {
    c := make(chan Result)
    go func() { c <- First(query, Web1, Web2) } ()
    go func() { c <- First(query, Image1, Image2) } ()
    go func() { c <- First(query, Video1, Video2) } ()
    timeout := time.After(80 * time.Millisecond)
    for i := 0; i < 3; i++ {
        select {
        case result := <-c:
            results = append(results, result)
        case <-timeout:
            fmt.Println("timed out")
            return
        }
    }
    return
}

The implementation shows Go’s concurrency primitives: goroutines, channels, and the select statement. The Go runtime manages the goroutines which are lightweight threads of execution. Goroutines communicate via channels. The switch statement allows merging values originating from multiple channels. These constructs are all built into the language, no external library is required.

Clojure

I was surprised when Rich Hickey mentioned the fake search example in his presentation on Clojure core async. I copied here the code from the slides for reference:

;; https://www.youtube.com/watch?v=drmNlZVkUeE?t=2458
(defn search [query]
  (let [c (chan)
        t (timeout 80)]
    (go (>! c (<! (fastest query web1 web2))))
    (go (>! c (<! (fastest query image1 image2))))
    (go (>! c (<! (fastest query video1 video2))))
    (go (loop [i 0
               ret []]
          (if (= i 3)
            ret
            (recur (inc i)
                   (conj ret (alt! [c t] ([v] v)))))))))

Clojure’s async library uses the same primitives as Go: the syntax is LISP, but the structure of the program is identical to that of the Go version. Watch Rich Hickey’s presentation if you’re interested how this works on the JVM and in a web browser using ClojureScript.

Haskell

My first implementation of the simulated search engine was in Haskell:

search30 :: SearchQuery -> IO ()
search30 query = do
    req <- timeout maxDelay $
        mapConcurrently (fastest query) [Web, Image, Video]
    printResults req

This is my favorite version of this exercise because it’s succint and expressive. We don’t see the primitives we saw in the Go and Clojure version, but an expression stating that some operations are expected to run concurrently. Writing this high-level code is possible because the threads are managed by the Haskell run-time system and I’m using the async library which exposes a powerful, composable API.

Rust

My latest addition to this collection is written in Rust, a language I’ve been learning for the last couple of weeks:

pub async fn search30(query: &SearchQuery) -> SearchResult {
    timeout(Duration::from_millis(80), async {
        tokio::join!(
            fastest(query, &SearchKind::Web),
            fastest(query, &SearchKind::Image),
            fastest(query, &SearchKind::Video),
        )
    })
    .await
    .map(|(web, image, video)| SearchResult::new(web, image, video))
    .unwrap_or_else(|_| SearchResult::timeout())
}

This code looks similar to the Haskell version because we don’t see channels and explicit thread management here either. The Rust language defines the async/await syntax and the related interfaces but it delegates the concrete execution strategy to external libraries. In this example I chose the Tokio library which is a mature asynchronous run-time library, but in the future I’d like to explore other libraries too and learn more about how they work.

Asyncronous programming in Rust is a recent addition to the language. If you’re interested how this feature was designed I recommend watching Steve Klabnik’s talk.

Summary

Modern languages provide ways of expressing concurrent operations using built-in language primitives or external libraries. Writing a simulated search engine is a great exercise to learn about concurrency because it requires to think about thread creation, thread cancellation and merging results from multiple threads.

Deploying thewagner.net

2020-07-03T00:00:00+02:00

For a long time I’ve been manually deploying this blog to GitHub Pages. This worked OK because I publish less than once a month. But I always wished for a better, automatic solution. Recently I used Nix to rebuild my home network and I was curious if I can improve the workflow of publishing my articles to thewagner.net.

Ingredients

I write the articles in Markdown files which are converted to static HTML files by Pelican. The source content is stored in a Git repository. I push to this repository when I write new content.

The blog is served from GitHub Pages so I don’t have to manage any servers. GitHub expects the HTML pages in a repository with a specific name: in my case this is wagdav.github.com. This repository only hosts generated content. I push to this repository when I want to publish the changes in the source repository.

I configured Travis CI to execute the build and deploy steps when a change is committed to the source repository.

In short:

The source repository thewagner.net stores articles.
The deployment repository wagdav.github.io stores generated HTML pages.
When triggered, Travis CI checks out the source repository, builds the blog and pushes the generated files to the deployment repository.

Let’s see the build and deploy steps in detail.

Build

The build pipeline is defined as a Nix expression. I don’t explain how it works, but I highlight the steps it performs:

Build the static website using Pelican.
Run static analysis on the bash scripts using shellcheck.
Run markdownlint to flag formatting issues in the articles.

These steps depend on tools from three different language ecosystems: Pelican is a Python project, shellcheck and markdownlint are written in Haskell and Ruby, respectively. Yet, the only dependency of running this pipeline is the Nix package manager.

To execute all the checks and build the blog I run:

nix build

All dependencies are pinned to a specific version of Nix Packages therefore this command always uses the same version of every tool and library no matter where or when it is executed.

Deploy

A shell script pushes the generated HTML files to the deployment repository:

./scripts/publish.sh

Because the script runs within a Nix shell, again, Nix is the only dependency and I don’t have to install any additional tools or libraries to run it.

Automate

The commands described in the previous sections can be executed on my local workstation. However, to deploy my changes reliably I choose Travis CI to automate the build, check and deployment steps.

The Nix support of Travis CI is fantastic: it takes only seven lines of sweet YAML to setup everything:

language: nix

deploy:
  provider: script
  script: nix-shell scripts/publish.sh $GITHUB_TOKEN
  on:
    branch: master

Behind the scenes this configuration builds the artifacts described in default.nix and executes the publish.sh when the change was triggered on the master branch. The secret value GITHUB_TOKEN is configured on the Travis CI web interface.

Summary

Deploying my blog is now fully automatic: I only push the contents of the articles to a repository. A hosted service builds and deploys the HTML pages. I can develop, test and execute each step of the pipeline locally and the dependency on hosted services is isolated to a few lines of code.

Homelab

2020-05-31T00:00:00+02:00

I own a few computers which I use to experiment with new languages, tools and technologies. I was dissatisfied with configuration management of these machines. I used to install and configure packages manually and I never remembered what I’ve changed. I tried Ansible but I found it tedious to maintain playbooks: it’s hard to remove all assumptions about the current state of the system you’re configuring and making all playbook tasks idempotent is close to impossible.

This article documents the current state of my home infrastructure which I configure primarily using Nix.

You can find all the configuration code on GitHub.

Routers

The external connectivity is provided by a DSL box of my Internet service provider (ISP). I change the default settings of this device as little as possible because, in my experience, these ISP provided boxes break or get replaced every other year.

When I have a problem with the box and I call the ISP’s support line they usually ask me to reboot the device. Then, because the reboot rarely solves anything, they ask me to perform a factory reset. When this happens all custom configuration from the box is gone and I have to start from scratch. I don’t want my home network setup to depend on specific features of the ISP-provided box.

I have a Linksys WRT3200ACM router connected to the ISP’s box which also provides WiFi for the apartment. The router runs OpenWRT and I use a script to modify the default settings: change the timezone, setup WiFi and DNS aliases, install the Prometheus OpenWRT node exporter.

NixOS servers

After a few weeks of exploration and learning I installed NixOS on all my computers at home, which include:

Old industrial PC (32-bit)
Intel NUC (64-bit)
Raspberry Pi 4 (64-bit ARM)
Thinkpad laptop (64-bit)

When it comes to deploying to physical hardware NixOS feels like the endgame. Because NixOS is designed from the ground up to be declarative I can store all the configuration of these machines in a Git repository. The deployments are atomic: if I break something I can roll back any change. I can create a virtual machine from an arbitrary machine configuration, test it locally, then deploy it to the real hardware. If I remove a service definition from the configuration files the services will be removed from the servers as well.

Previously I tried managing servers using Salt and Ansible and I’m never looking back.

Let me demonstrate with an example the level of composability Nix enables. The core part of the Nix module that configures Prometheus on one of my nodes reads like this:

services.prometheus = {
  enable = true;
  scrapeConfigs = ...;
};

services.consul.catalog = [
  {
    name = "prometheus";
    port = 9090;
  }
];

networking.firewall.allowedTCPPorts = [ 9090 ];

This snippet adjusts three separate components:

Enable and configure Prometheus
Create an entry in the Consul Service Catalog for service discovery
Open a port in the firewall

A typical service often relies on other services, monitoring agents, network configurations and other tools. However, these dependencies are hard to express in traditional configuration management tools. In NixOS they are described at the same place using a single, unified syntax.

The complete module also contains the full Prometheus configuration, the scrapeConfigs attribute, which I elided here.

Sensors

I installed temperature and humidity sensors in two rooms and a few smart switches. I use Nix to create the development environment for flashing the firmware and for provisioning these embedded devices.

Also, I wrote a small Nix module to make the sensor configuration more expressive. For example, instead of a cryptic JSON document:

{
  "NAME": "ZJ-ESP-IR-B-v2.3",
  "GPIO": [0,0,0,0,51,37,0,0,39,38,0,0,0],
  "FLAG":0,
  "BASE":18
}

the GPIO ports of a LED controller equipped with an infrared receiver are assigned like this:

tasmota.template {
  name = "ZJ-ESP-IR-B-v2.3";
  gpio = with tasmota.component; {
    GPIO4  = IRrecv;
    GPIO5  = PWM1;
    GPIO12 = PWM3;
    GPIO13 = PWM2;
  };
};

The sensors publish their data over MQTT. Then, a Telegraf service exports the MQTT messages for Prometheus. Finally, the Prometheus time series are displayed in Grafana dashboards. There are many moving pieces here, but the resulting configuration is short and readable: for example the module setting up MQTT-Prometheus conversion is only 66 lines long.

Summary

Today I run the following services on my home computers:

Consul and Consul Template: Service discovery and dynamic service configuration
Gitolite: Host private Git repositories
Grafana: Display metrics on dashboards
Mosquitto MQTT broker: relay messages from the sensors
Nginx: HTTP server and reverse proxy
Prometheus and its exporters: Collect metrics
Telegraf: Export MQTT sensor data as Prometheus metrics

In the future I’m planning to configure WireGuard VPN access and experiment with network booting and periodic, automatic reinstalling of NixOS on my servers.

Overall I’m very pleased with this setup. NixOS allows fearless tinkering in my homelab. Deploying from a Git repository is easy and requires minimal maintenance. I can reconfigure or even reinstall my whole network within minutes.

Acknowledgment

I’m thankful to Alessandro Degano for suggesting me Telegraf and showing me how to set it up.

Exploring Nix

2020-04-30T00:00:00+02:00

For the last few weeks I’ve been exploring NixOS and its related tools. This article is an experience report and a collection of learning material I find useful.

NixOS is a unique Linux distribution with origins in research conducted at Utrecht University. NixOS handles software delivery and configuration in a different way than any other distribution I know. The entire operating system is treated as an immutable value which makes deploying and maintaining NixOS-based systems easy and reliable.

The language

NixOS is configured using the Nix expression language: a small, purely functional programming language. Besides the usual data types (booleans, numbers, strings, lists and sets) Nix has some features which are uncommon in configuration languages.

Let blocks bind values to symbols. The bindings appear after the keyword let and the symbols can be used after the keyword in. For example, the expression:

let
  x = "a";
  y = "b";
in
  x + y + x

evaluates to "aba".

In Nix, functions are first-class values. A function which adds one to a value is defined succinctly:

n: n + 1

Typically, functions take sets as arguments and return sets as results. This expression defines and calls the endpoints function:

let
  endpoints = { engine, domain }: {
    http = "http://${engine}.${domain}";
    https = "https://${engine}.${domain}";
  };
in
  endpoints { engine = "google"; domain = "com"; }

which evaluates to:

{ http = "http://google.com"; https = "https://google.com"; }

This last example also shows how symbols in the current scope are interpolated in strings using ${}.

These constructs are sufficient to read the examples in this article. For a comprehensive overview of the language features see this tutorial.

Derivations

A derivation, a core concept of Nix, is a build recipe: it describes how to obtain, in other words derive, a component from its inputs. Let’s make this statement more concrete with an example.

We will use Nix to put a string into a file using a build action equivalent to:

echo hello from Nix > output.txt

This build runs the echo shell command to generate a file. The build requires no input source files, but a shell to be present. We could use bash but from where do we get its executable?

Typical build systems assume that certain programs are available in the build environment. Nix doesn’t make such assumptions. Builds are performed in isolation: no programs, no environment variables, nor access to the outside world are available unless explicitly specified.

In programming we use a function to abstract over an input parameter of a computation. Let’s do the same and write a function which takes bash as input and returns the build recipe, a derivation:

# hello.nix
{ bash } :

derivation {
  name = "hello";
  builder = "${bash}/bin/bash";
  args = [ "-c" "echo hello from Nix > $out" ];
  system = builtins.currentSystem;
}

derivation is keyword of the Nix language. It’s represented as a set with specific attributes: a name, a program and its arguments to produce some output, and a specification of the operating system’s architecture where this derivation can be built.

The bash argument is a dependency, it refers to a derivation which must be built before the hello derivation.

To understand better the derivation’s structure, let’s assume we have bash built and we evaluate the hello derivation. Nix stores the resulting derivation in the following structure:

{
  "/nix/store/61lcv6k65f42d3v8nww7m7k48h7v9mhy-hello.drv": {
    "outputs": {
      "out": {
        "path": "/nix/store/qcnf97fclrnqppq3h5ld9smqdb8l2ybk-hello"
      }
    },
    "inputSrcs": [],
    "inputDrvs": {
      "/nix/store/8ynv2wxv6vaa75sbpmz8rnlbv1bxcfzn-bash-4.4-p23.drv": [
        "out"
      ]
    },
    "platform": "x86_64-linux",
    "builder": "/nix/store/9si14qjcz8072c0v42zbkglq08s2cg04-bash-4.4-p23/bin/bash",
    "args": [
      "-c",
      "echo hello from Nix > $out"
    ],
    "env": {
      "builder": "/nix/store/9si14qjcz8072c0v42zbkglq08s2cg04-bash-4.4-p23/bin/bash",
      "name": "hello",
      "out": "/nix/store/qcnf97fclrnqppq3h5ld9smqdb8l2ybk-hello",
      "system": "x86_64-linux"
    }
  }
}

In this representation we can see how Nix stores the build artifacts under /nix/store. The filenames in the store are cryptographic hashes of the defining Nix expression suffixed with a readable, user-provided name.

This data structure is a concrete description how to build the hello greeting:

Outputs: the derivation yields one output named out. The output will be stored under the given path.
InputSrcs: the derivation requires no input sources
InputDrvs: the derivation depends on the derivation which builds bash.
Platform: on my PC the expression builtin.currentSystem evaluates to x86_64_linux.
Builder and args: the program and its arguments we provided, but now referencing concrete artifacts in the store.
Env: the environment variables available during build. These are originating from the derivation’s attributes we provided in the Nix expression. We reference $out in the build script.

Derivations are just data without any knowledge of the Nix language. When evaluated, a Nix expression typically produces many, interdependent derivations which are built in topological order, that is each derivation is built after their dependent derivations.

Nix builds a component in two stages:

Evaluates the expression and writes resulting derivations in the Nix store.
Builds the derivation and writes build results in the Nix store.

Nix expressions provide a high-level language for developers to define components, component configuration and component relationships. Derivations encode build instructions for a single component for a given configuration in a well-defined environment.

Nix expressions can be evaluated on any system where Nix is installed. Derivations may be copied to other nodes for building: to a build farm, or to nodes with special hardware.

Build the example

To build the example in the previous section, save the expression in a file named hello.nix and run:

$ nix-build hello.nix --arg bash '(import <nixpkgs> {}).bash'
/nix/store/qcnf97fclrnqppq3h5ld9smqdb8l2ybk-hello

The --arg switch tells Nix to use the bash package from the Nix Packages collection. nix-build prints the path where the build result is stored. By default, nix-build also creates a symbolic link to the build result in the current working directory so it’s easy to verify if we find the string we expect:

$ < result
hello from Nix

To see the internal structure of the derivation, run the command:

nix show-derivation /nix/store/qcnf97fclrnqppq3h5ld9smqdb8l2ybk-hello

which outputs data structure as shown in the previous section.

Composing an operating system

The Nix language provides a clean component description formalism: a single expression builds 40000 packages on multiple platforms. To achieve this scale, higher-level abstractions are built from the Nix language primitives.

Modules are functions which return a set with specific attributes:

{ config, pkgs, ... }:

{
  imports = [
    # paths to other modules
  ];

  options = {
    # option declarations
  };

  config = {
    # option definitions
  };
}

Modules are useful for configuring complete subsystems such as networking, printing, graphics and so on. The top-level NixOS configuration, typically stored in /etc/nixos/configuration.nix, is a module itself which may include other modules.

For example, the firewall module allows you to specify the open ports of your system:

networking.firewall.allowedTCPPorts = [ 80 ];

The module keeps the intricacies of generating iptables rules within its implementation. Our system configuration remains simple and declarative.

Applications

The existence of NixOS proves that the Nix expression language is a solid foundation for software packaging and software delivery. NixOS is not the most popular Linux distribution today, but if you felt the pain of server management using tools like Puppet, Salt or Ansible, you should definitely give NixOS a try.

You don’t have to replace your operating system to try Nix. You can use Nix packages on Linux and Mac systems to set up and share build environments for your projects, regardless of what programming languages and tools you’re using.

The Nix model can also be used to deploy servers and for continuous integration and delivery.

Learn more

This section is a collection of online resources which I find useful to learn about Nix.

Learn the Nix language:

Learn about modules and overlays:

Browse Nix packages and NixOS options:

Read about the original research:

Integrating Software Construction and Software Deployment (Nix used to be called Maak)
The Purely Functional Software Deployment Model

Additionally, the NixOS Wiki contains a list of learning resources.

Summary

After a few weeks of learning I’m amazed by Nix. I believe NixOS is the best operating system to deploy from a declarative configuration. The expression language, the Nix store, the evaluation and build strategy were built for painless software deployments.

If you value infrastructure as code and immutable deployments you should definitely spend time on Nix and its core concepts.

I updated my main laptop and my servers at home to NixOS. The configuration of all my machines is available on GitHub.

Parallel mindset

2020-02-29T00:00:00+01:00

Perhaps everything in our mainstream programming languages is at least 50 years old. Loops, iterators, pointers and references to mutable memory locations appeared in Fortran or ALGOL and now they are part of all mainstream programming languages.

These constructs were designed for developing sequential programs. But today computers have many cores and processors and we want to do more computations in parallel. We bolted some parallel features on our sequential programming languages, but sequentiality remains built into our mindset and our infrastructure.

We need to change our mindset.

Automate parallelism

Many programming languages manage allocations of storage automatically. Instead of the programmer, the compiler or the run-time system claims and frees memory. The implementation of this automatism is language specific:

Scope-limited automatic variables (C++)
Tracing garbage collection (Java, Python, Go, Haskell)
Ownership tracking (Rust)

Would it be possible to make parallelism, that is allocation of code to processors, automatic?

With automatic memory management we stopped using malloc and free. To automate parallelism we need to restrict our programming style and give up sequential programming language artifacts from the sixties.

Accidentally complex

Let’s take the textbook example of computing $n!$:

def factorial(n):
    result = 1                  # ①
    for i in range(1, n + 1):   # ②
        result = result * i     # ③
    return result

This code is written today’s most common programming style: sequential, imperative, mutable and allowing uncontrolled side effects. Every line of this function states what happens during execution:

Assign an initial value to result.
Use the value of i drawn from the specified range.
Mutate result with the current value of i.

Many consider this implementation “readable” and “simple” because we are used to seeing such programs. But this code contains many accidental aspects: the result accumulator, the intermediate value i, and the allusion to sequential execution. These are unrelated to the original problem statement. This is a form of complexity caused by control.

Keeping the essential

Let’s strip off everything but the essential from the previous implementation of factorial:

from functools import reduce
from operator import mul

def factorial(n):
    return reduce(mul, range(1, n + 1))

This implementation reads almost as a pure specification without any view on the execution. This form is more unusual but simpler than before because all accidental complexity were removed.

As programmers, we give up the control of the execution order and we let the runtime environment choose the most efficient execution strategy. In case of Python the execution would be similar, or perhaps identical to that of the first implementation. Using this declarative style, however, we could imagine a programming system where the actual execution strategy would depend on multiple factors:

For small n execute sequentially.
For large n split the range among multiple processors, then merge the results.
Push some parts of the computation to the GPU.
Send the computation to a massively parallel super-computer.

In general, operations cannot be parallelized arbitrarily. To allow for such flexible runtime behavior we must enrich our programs with hints to the compiler or to the runtime environment.

Algebraic properties

If we recognize and communicate our problem’s algebraic properties to the compiler or to the run-time, it can exploit alternate representations and implementations.

Well-known algebraic properties translate to useful hints:

Associative: grouping doesn’t matter
Commutative: order doesn’t matter
Idempotent: duplicates don’t matter
Identity: the current value doesn’t matter
Zero: other values don’t matter

In factorial the integer multiplication is associative, therefore performing the multiplication in groups first, then merging the partial results is a correct parallel implementation. It is also commutative, so we can do the merging in any order.

Automatic parallelism in practice

Optimizing compilers generate efficient, often parallel code from a sequential, imperative code. But in general, a program organized according to linear problem decomposition principles is hard to parallelize. Frances Allen won the 2006 Turing-award for her work in program optimization and parallelization.

Dask is a flexible parallel computing library for Python. Its user interface mimics the programming experience of popular data processing libraries. For example, you write regular NumPy array manipulation code and the Dask scheduler distributes the computations across multiple threads or among the nodes of a cluster.

Facebook’s Haxl library can automatically execute independent data fetching operations concurrently. Haxl, with the compiler’s assistance, recognizes algebraic properties such as applicative functor and monad to generate efficient concurrent code. Simon Marlow’s presentation is a great introduction of the ideas behind this tool.

Summary

When we code in imperative style, accidental complexity of control creeps into our programs. The programmer, instead of the expressing problem’s essence, is burdened with managing loops, states and the details of a sequential-looking runtime behavior.

Code written in declarative, functional style with no ties to a specific execution model may be automatically parallelized by the underlying system. Algebraic properties constrain which execution strategies are correct and efficient.

The inspiration to this article came from the talk “Four Solution to a Trivial Problem” of Guy Steele. I recommend watching the whole talk on YouTube.

I also recommend reading Bartosz Milewski’s post on Parallel Programming with Hints

The essence of a CI/CD pipeline

2020-01-07T00:00:00+01:00

Practitioners of continuous integration often describe the process of automated software delivery as a pipeline: the source code enters the pipe, it is compiled, tested, packaged and released product comes out on the other end.

This methaphor evokes the notions of delivering, modularity and continuity. Teams of different backgrounds relate to this image even without understanding each transformation step.

But what is a software delivery pipeline? In this post, instead of a metaphor, I propose a precise mathematical model of it.

Concept zoo

I reviewed five popular CI/CD systems where users model their software delivery process by defining a pipeline:

Let’s see how the relevant user documentation describe the pipeline and its related concepts.

Task, action, step

The reviewed systems call the pipeline’s unit of work task, action or step.

Azure Pipelines

A step is the smallest building block of a pipeline. A step can either be a script or a task.

CircleCI

A step is an executable command.

Concourse

A task is the smallest configurable unit. A task can be thought of as a function from inputs to outputs that can either succeed or fail.

GitHub Actions

Individual tasks that you combine as steps to create a job. Actions are the smallest portable building block of a workflow.

GoCD

A build task is an action that needs to be performed. Usually, it is a single command.

The names differ but they all describe a similar concept: the unit of work is an executable script or command.

Concourse’s task definition proposes a precise semantic model: a task is a function. We will build on this model later.

Job

A job is an ensemble of tasks, actions or steps.

Azure Pipelines

A job represents an execution boundary of a set of steps. All of the steps run together on the same agent.

CircleCI

Jobs are collections of steps.

Concourse

Jobs are sequences steps to execute.

GitHub Actions

A defined task made up of steps. Each step in a job executes in the same runner.

GoCD

A job consists of multiple tasks, each of which will be run in order.

At this concept the definitions start to diverge, still there are some common points:

Actions, tasks or steps build up jobs.
A job’s components usually run sequentially.
A job’s components usually run on the same build agent, executor or runner.

The notable exceptions are:

In Concourse it’s possible to run a job’s steps in parallel.
In Concourse and GoCD there are no locality guarantees on where the job’s tasks are run.

These job definitions are operational and not denotational: instead of defining what a job means they focus on how a job is executed.

Stage

In some systems jobs can be grouped into a stage.

Azure Pipelines

A stage is a logical boundary in the pipeline. Each stage contains one or more jobs.

GoCD

A stage consists of multiple jobs, each of which can run independently of the others.

Azure Pipelines runs the stages sequentially by default, but arbitrary ordering can also be defined between them. This includes no ordering at all, meaning that stages can run concurrently.

CircleCI, Concourse and GitHub Actions don’t have this concept.

Pipeline, workflow

We now are ready to define a pipeline, also called workflow.

Azure Pipelines

A pipeline defines the continuous integration and deployment process for your app. It’s made up of one or more stages.

CircleCI

Workflows define a list of jobs and their run order.

Concourse

Pipelines are built around jobs and resources. They represent a dependency flow.

GitHub Actions

Workflows are made up of one or more jobs and can be scheduled or activated by an event.

GoCD

A pipeline consists of multiple stages, each of which will be run in order.

Jobs or stages are grouped into pipelines. The definitions are again operational with an emphasis of execution order and dependencies.

Pipeline, simplified

Now we’ve seen some of the concepts of the most popular CI/CD systems. Some systems have even more which I didn’t cover here.

Do we need all these to model the software deliver process?

Task as a function

Let’s revisit Concourse’s task definition: A task can be thought of as a function from inputs to outputs that can either succeed or fail.

This is a great definition because it specifies what a task means and not what it does or how it does it. Developers can choose to implement a task as they deem most fitting but the user can think of it as a function no matter what.

Let’s see some task examples:

A compilation task takes a source code as input and produces a compiled binary as output.
A test task takes the compiled binary as input and produces a test report as output.
A release task takes the compiled binary and a test report. If the test report is acceptable (no tests fail, test coverage is OK) it releases the binary and returns a link to repository where the software can be downloaded.

I named the unit of work “task”. As we’ve seen other systems prefer “step” or “action”, which would be totally fine as well.

Let’s write down formally Concourse’s task definition.

type Task a b = a -> Maybe b

A Task is a function with two type parameters a and b representing its input and output types, respectively. To express possible failure, the output is wrapped in Haskell’s Maybe type. In other languages this is called Option, optional or Result.

I wrote down this definition in Haskell’s syntax but this is not important. What matters is that our model, the Task’s meaning, is mathematical function.

These are the type signatures of the tasks described previously in words:

--         input type        output type
build   :: SourceCode     -> Maybe CompiledBinary
test    :: CompiledBinary -> Maybe TestReport
release :: CompiledBinary
        -> TestReport     -> Maybe PackageURL

These are not the implementation of these tasks but their definition expressed as Haskell code.

Sequential composition

Let’s define a task to tests the incoming pull requests of our project.

This task takes the pull request’s source code, builds the binary, runs the tests and returns the test report. The test report is for the reviewers to judge the quality of the proposed change.

In short, we want sequence the tasks build and test. If we had an operator with this type signature:

inSequence :: a -> Maybe b -- first task
           -> b -> Maybe c -- second task
           -> a -> Maybe c

we could express the pull request validating task as:

validatePullRequests :: SourceCode -> Maybe TestReport
validatePullRequests = inSequence build test

where

validatePullRequests is a Task because it’s a function with the right type signature
The source code is fed to the first task, build
The resulting type of build is Maybe CompiledBinary
If build fails the result of the whole task is failure
Otherwise, feed the compiled binary to test

I haven’t shown you the definition of inSequence, but you can verify that in the expression validatePullRequests the types match. You can also see that inSequence looks almost like regular function composition except the output types are wrapped in Maybe.

Parallel composition

Let’s consider now two independent tasks:

unitTests        :: SourceCode -> Maybe UnitTestReport
integrationTests :: SourceCode -> Maybe IntegrationTestReport

These two test suites could be run in parallel, because they both only depend on the SourceCode value.

We don’t want to introduce a new concept, but we want the result of parallel composition to be a Task as well. We’re after an operator with the following type signature:

inParallel :: (a -> Maybe b)
           -> (a -> Maybe c)
           -> (a -> Maybe (b, c))

The composite task yields the results of input tasks as a tuple. If any of the two task fails, the result of the composite task is failure (represented by the value Nothing).

Using inParallel we could write a task to run all tests:

runAllTests :: SourceCode
            -> Maybe (UnitTestReport, IntegrationTestReport)
runAllTests = inParallel unitTests integrationTests

The inParallel operator represents a “fan-out” structure in the pipeline where independent transformation steps are applied on the same input.

Semantic model

In the previous sections we’ve defined a denotational model for CI/CD build tasks:

type Task a b = a -> Maybe b

which maps the Task concept to its meaning, a mathematical object. This serves not only as a mental model, but also it allows us introduce regular and powerful composition rules.

I’ve shown you inSequence and inParallel combinators. For reference, without explanation, here are their definitions:

inSequence :: Task a b -> Task b c -> Task a c
inSequence t1 t2 x = t1 x >>= t2
-- or equivalently
                   = t1 >=> t2

inParallel :: Task a b -> Task a c -> Task a (b, c)
inParallel t1 t2 x = liftA2 (,) (t1 x) (t2 x)

These combinators are expressed using the task’s semantic model without operational terms or unnecessary limiting assumptions.

It turns out that inSequence and inParallel are not primitive operations. Tasks and their composition rules can be defined using a more general vocabulary of arrows. This suggests that the semantic model is powerful enough to model any software delivery process.

Using this model, jobs, stages, workflows and pipelines are just Tasks.

Summary

Today’s popular CI/CD systems are built around the metaphor and not a rigorous definition of a pipeline. I propose Maybe-valued functions as a semantic model for a build task. Using well-studied and precisely defined rules, tasks can be composed to model the software delivery process.

In a future post I will present an experimental system which uses these principles to express continuous integration and continuous delivery pipelines.

Acknowledgment

Many thanks to the members of the Pix4D CI team for the inspirational discussions during coffee breaks.

I’m grateful to Conal Elliott for reviewing an early draft of this article and for providing valuable feedback.

Functions in disguise

2019-12-20T00:00:00+01:00

Mainstream programming languages support structured programming. Functions, subroutines and methods make code more expressive and reduce repetition.

Pure functions, constructs that produce values by using only their input arguments and nothing else, are of particular importance.

Pure functions express intent explicitly because, by definition, they don’t rely on side effects. Pure functions are easier to test and reason about than those which have observable side-effects. This post is only about pure functions, so I’m dropping the “pure” qualifier.

In functional programming programs are composed of expressions and declarations of functions. In this niche domain the utility of functions is well studied and understood. Ideas from purely functional programming languages percolate into mainstream programming languages, but functional programming is far from mainstream.

There is, however, a form of functional programming practiced by every developer every day. They write configuration files. This is the hardest form of functional programming where functions are not used at all.

Functional configuration

Every non-trivial software requires some configuration. When a program reads its configuration it executes a small, often trivial functional program. Let me show you what I mean with an example.

Consider the following, INI-style, configuration file:

[staging]
url=staging.example.com

[production]
url=example.com

Let’s say that this is a configuration file of a simple service which is deployed into a specific environment. First, we deploy it to staging and, when all the tests pass, we promote the application to production. Naturally, the service’s address is different in these two environments.

This is pretty standard, we see similar configuration blocks everywhere. Where’s the functional programming here?

With the configuration blocks we implicitly created a simple function. Its imaginary type signature is:

configuration :: Environment -> Url

This reads: the configuration is a function from an environment to a URL. If we saw such a function signature in an application, we could implement it like this:

def configuration(environment):
   return {
     url: 'example.com' if environment == 'production' else 'staging.example.com'
   }

For clarity I used Python’s familiar syntax, but this is not important.

Conceptually we can think that after the configuration file is parsed, this function is evaluated. In practice, we usually don’t see this function written out, but it’s hidden somewhere between the configuration format parser library and our application’s initialization.

What we see is that a function, a universally useful concept in programming, is not being used explicitly to define the program’s configuration.

But the configuration is simple

Let’s continue on the previous example and add more configuration parameters:

[staging]
url=staging.example.com
db_backend=postres
db_address=%(db_backend).db.example.com

[production]
url=example.com
db_backend=rds
db_address=%(db_backend).amazon.com

Here I made the database backend of our service configurable. It’s a contrived example, but not completely unrealistic: in staging we use our own Postgres database engine as opposed to production where we wish to store our data in Amazon’s database-as-service offering called Relational Database Service (RDS).

Note that I reused the value of db_backend in the definition of db_address because I wanted to avoid duplication. You can parse this configuration file with Python’s configparser module.

Configuration formats with sections and custom interpolation rules are very common. Still, there are some interesting questions to ask:

What is a valid section name?
What happens if I repeat the same entry in a section?
What happens if I omit an entry in a section?
What are the interpolation rules of the percent expressions?

In this specific case, we find the answers in the not so short documentation, but the point is that a concept like ‘configuration block’ looks deceptively simple. It takes quite some effort to precisely explain what a file like this means.

Now let’s see how this configuration file looks if we represent it explicitly as a function:

def configuration(environment):
  if environment == 'production':
    url = 'example.com'
    db_backend = 'postgres'
    db_suffix = '.db.example.com'
  else:
    url = 'staging.example.com'
    db_backend = 'rds'
    db_suffix = '.amazon.com'

  return {
    url: url,
    db_backend: db_backend
    db_address: db_backend + db_suffix
  }

Is this better than the INI syntax? Well, not necessarily, but it triggers different kinds of thoughts:

Should we avoid repeating the example.com domain?
Should we introduce a richer data type to represent the database configuration?
Should we split the database configuration in a separate function?

The answers to these questions depend on the specific context. Note, however, that these questions are programming questions. We raise similar questions when we write the core of our applications. Why doesn’t configuration deserve the same level of scrutiny?

Again, I gave the example in Python’s syntax for simplicity, but the configuration language doesn’t have to be Python, but it could support function definitions.

Functions hide everywhere

Let’s leave our toy example and look for functions elsewhere.

Packer

Packer is a tool for building machine images. Its documentation explains how to write configuration templates and use variables to further specialize them from the command line.

The appearance of the word ‘variable’ suggests that some function-related business may be going on here. Indeed, Packer templates are functions from user variables to build instructions. Interestingly, Packer templates themselves can call functions too.

Now Packer is a wonderful tool and I’m not claiming that there’s anything wrong with it. I just want you to realize that in a Packer configuration there lies an ad hoc, mini functional program.

Rolling your own implicitly functional configuration language is a lot of work and it develops some warts. For example, there are restrictions where you can or cannot use variables in a Packer template. Again, this is not a problem, you can use Packer just fine. It’s just a pity because variable scoping is pretty well understood since the development of ALGOL in the 1950s.

Terraform

Terraform allows you to define your cloud infrastructure as code. I use Terraform every day and I cannot imagine my work without it.

Terraform’s configuration language already supports various programming constructs, but remains implicitly functional. You cannot define a function explicitly, but a “module”. Just observe the language used to describe them. These are quotes from the documentation, the emphasis is mine:

Input variables to accept values from the calling module.
Output values to return results to the calling module.
To call a module means to include the contents of that module into the configuration with specific values for its input variables.

It’s no surprise that modules actually are functions, or they should be.

You could imagine that when you import a module you compute a value which contains a bunch of other functions, those that are defined in the module. In fact, this is exactly how JavaScript modules worked before the language standard introduced the import keyword.

Ansible and Salt

Configuration management systems such as Ansible and Salt allow you to manage a large number of machines. You specify your servers’ configuration in a YAML file and these systems make sure that the desired files, software, etc., are deployed on them.

A core feature of these tools is the ability to specify configuration file templates which are rendered using context specific parameters (for example: the server’s assigned IP address). These templates are functions which take user parameters as arguments and return the rendered configuration file as a string.

Salt and Ansible also support structuring your configuration into separate files. Sadly they are not called modules, but roles and formulas.

If you take a look at the documentation of Ansible roles you won’t be surprised to find word “variable” again. The second paragraph of Ansible’s documentation claims: Ansible’s main goals are simplicity and ease of use. Take a look how “simple” it is to use variables.

Warm and fuzzy names

What’s wrong with the names “template”, “role” and “formula”? Just replace all their occurrences with the word “function” and we’re done. Well, no. It’s not at all about linguistics.

The problem is that your preferred warm and fuzzy name is inevitably more ambiguous than the concept of a function. When you refuse to admit that you’re manipulating functions in your software configuration you’re throwing away 60+ years worth of computer science. Additionally, you’re creating extra work for yourself to come up with some idiosyncratic rules for your users to write modular and comprehensible configuration files for your system.

Why not use functions as the core abstraction to build up configuration files? Functions can be composed using rigorous, well-defined mathematical rules. A functional configuration language could provide few, but powerful concepts to author configuration files for complex software systems.

Of course this not to say just use Maths and you’re done. There’s a lot more to a good configuration language than that. Nevertheless, I firmly believe that it’s possible to provide developer ergonomics, readability on top of solid concepts borrowed from Mathematics and Computer Science.

Summary

Today, configuration files are written in ad hoc, implicitly functional programming languages where functions don’t appear explicitly but disguised. This leads to the proliferation of ill-defined concepts and idiosyncratic rules.

This makes extremely valuable tools, like those mentioned in this article and many others, unnecessarily hard to understand, configure and use.

It is possible to define a disciplined, functional programming language for configuration files. I encourage you to take a look at Dhall. Get over its unusual syntax and devote some time understanding the fundamental role of functions in software configuration and in programming in general.

Acknowledgment

I’m grateful to Kyle M. Douglass for reviewing an early draft of this article and for providing valuable feedback.

O’Reilly Velocity Conference 2019

2019-11-08T00:00:00+01:00

Between November 5-7 I spent three days on the O’Reilly Velocity conference. This post documents the sessions I liked the most.

The conference organizers collected the speaker slides here

Tuesday

I travelled to Berlin on Monday and started the conference with two, half-day tutorials.

Open Telemetry workshop

In this workshop, tutored by Liz Fong-Jones, I learned about OpenTelemetry: a project aiming to standardize metrics and trace collections from applications.

Our task was to instrument the provided Go code. This was an HTTP service computing the Fibonacci sequence using its recursive definition. To compute the preceding two numbers in the sequence the server code repeatedly issues HTTP client calls to itself. This implementation is terrible for production, but it is an excellent classroom example.

For the practical exercises we wrote the code on Glitch. I think this was a great choice, I could immediately start working on the execrcises without installing anything on my computer.

SRE classroom (PubSub)

In this workshop Google engineers lead us to design a strongly consistent, multi-datacenter queue. The instructors described how the system should behave, what latencies are tolerated, what failures modes are allowed. We formed groups of five-six people to come-up with an implementation plan. Then, we created a bill of materials to estimate the cost of building the system.

Wednesday

The power of good abstractions in systems design Lorenzo Saino (Fastly)

slides
speaker’s homepage
90% of the problems are solved with just 40 basic principles
TRIZ - theory of inventive problem solving

Kubernetes the very hard way Laurent Bemaille (Datadog)

Privilege escalation in build pipelines Andreas Sieferlinger (Scout24)

slides

Grafana and metrics without the hype and anti-patterns (wasn’t that good finally) Björn Rabenstein (Grafana Labs)

slides

Configuration is riskier than code Jamie Wilkinson (Google)

event page

M3 and Prometheus Rob Skillington (Chronosphere), Łukasz Szczęsny (M3)

Thursday

Building high-performing engineering teams Lena Reinhard (CircleCI)

Controlled chaos: devops and security Kelly Shortridge (Capsule8)

slides
recording
The “DIE” triad: distributed, immutable, and ephemeral infrastructure

Cultivating production excellence Liz Fong-Jones (Honeycomb)

Keptn: Don’t let your deliver pipelines become your next legacy code (Dynatrace Software)

Keptn

Stateful systems in the time of orchestrators Danielle Lancashire (Hashicorp)

Stateful workloads in Nomad

Consensus is for everyone Tess Rinearson (Tendermint Core)

Polymorphism and testing

2019-10-28T00:00:00+01:00

Recently I contributed a retry functionality to the cache-s3 Haskell project. After I had shared a draft PR with Alexey Kuleshevich, the project’s maintainer, he suggested some improvements. In this post I describe a simplified version of the resulting upstream pull-request and explain how it works.

Retrying in case of failure

Retrying is a simple error recovery algorithm. Communicating software components regularly experience network failures. If the network outage is intermittent, retransmitting the data shortly after a failing transfer may succeed. Thus, reducing the apparent error rate of the subsystem.

cache-s3 is used by continuous integration (CI) systems for storing a compilation cache database in an S3 bucket. Typically, a successful build is followed by the upload of the cache database, so that it can be reused by the next build. If the database upload fails the CI system fails the whole build task. This is annoying when you waited a long time for your build, but now you have no build and nor a saved cache database. You have to start everything again…

Instead, if the upload to the bucket is retried a couple of times the intermittent network problem is completely hidden from the user.

Give me an integer

In cache-s3 a complex HTTP request is retried, in this post I’m using a simpler example. The action we will retry is a question: we ask the user for an integer:

question :: IO (Maybe Int)
question = do
  putStr "Give me an integer: "
  readMaybe <$> getLine

This function returns Nothing if the user enters anything but an integer.

GHCI> question
Give me an integer: 5
Just 5

GHCI> question
Give me an integer: FOO
Nothing

If this question is a part of a longer quiz we let the user guess again before we fail the whole program. We would like to write a function retry with the following type signature:

retry :: Int -> IO (Maybe a) -> IO (Maybe a)

This function takes two arguments: the number of times to retry and the action to run. It returns a more perseverant action with the retry logic “baked-in”. In other words the retry alters the action’s runtime behavior, but it doesn’t change its type.

retry will operate like this:

GHCI> retry 3 question
Give me an integer: text
Retrying 1/3
Give me an integer: 5.5
Retrying 2/3
Give me an integer: 5
Just 5

In this example, after the second additional attempt the integer’s was finally received.

Implementing retry

Here’s an implementation of retry:

retry
  :: Int -- ^ number of times to retry
  -> IO (Maybe a) -- ^ action to retry
  -> IO (Maybe a)
retry n action = action >>= go 1                             -- ①
  where
    go i (Just res) = return (Just res)                      -- ②
    go i Nothing                                             -- ③
      if i > n
        then return Nothing                                  -- ④
        else do
          putStrLn $ "Retrying " <> show i <> "/" <> show n  -- ⑤
          res' <- action                                     -- ⑥
          go (i + 1) res'                                    -- ⑦

This function uses a recursive inner function, called go by convention.

The provided action is executed and its result is passed to go
The action succeeded, just return its result
The action failed, try again
If limit is reached we return Nothing
Otherwise inform the user about the retry in progress
Re-execute the action
Pass the result to the next iteration of go

In real code we’d wait a bit before step ⑥ , but now I’m skipping this. You can load this function into an interactive session and try how it works. You should see an output similar to that in the previous section.

Notice that retry is polymorphic in the action’s return type a. In the function’s body we don’t manipulate this value at all, therefore a can stand for any type.

Let’s develop an automatic test for this function and convince ourselves and our colleagues that this function really does what it’s supposed to do. The bad news is that in its current shape this function is hard to test because it’s doing too much IO operations.

Next, we are going to make retry more polymorphic and more testable.

Make it more polymorphic

The problem with the first implementation is the appearance of IO. We need to get rid of that. Notice that in the function’s body we don’t do abitrary IO operation but really just calling putStrLn as a logging function. The appearance of the bind (>>=) operator tells us that we exploit that IO is a monad.

Inspired by these two observations we replace IO with a generic m type constructor with two constraints:

retry :: (Monad m, HasLogFunc m) => Int -> m (Maybe a) -> m (Maybe a)

The Monad typeclass is part of the Prelude (“built-in”). We define the HasLogFunc class ourselves:

class HasLogFunc m where
  logInfo :: String -> m ()

instance HasLogFunc IO where
  logInfo = putStrLn

The first two lines defines the HasLogFunc as an interface where logInfo must be implemented. The last two lines provide IO with an instance of this interface: the implementation of logInfo is just putStrLn.

With the following modifications we obtain to a more generic form of retry:

retry ::
     (Monad m, HasLogFunc m)                                -- ①
  => Int         -- ^ number of times to retry
  -> m (Maybe a) -- ^ action to retry                       -- ②
  -> m (Maybe a)

-- same code as before

          logInfo $ "Retrying " <> show i <> "/" <> show n  -- ③

-- same code as before

Two constraints restrict what m can be: it must be a monad and must have a log function
Instead of IO-only, the function operates on generic actions of m
We replace putStrLn with logInfo which is available in HasLogFunc environment

This version works exactly the same as the first attempt because IO is a monad and we took care of its HasLogFunc instance.

More polymorphic, more testable

This new version of retry is more testable because in our test code we are free to use any m (given it satisfies the constraints we imposed) to express our assertions. To demonstrate this let’s test if the right sequence of error messages are displayed to the user. For example, if we had an action which always fails:

alwaysFails :: FakeAction (Maybe String)
alwaysFails = return Nothing

We’d expect retries until the specified number of attempts is exhausted.

We want our tests to be pure therefore, instead of using IO, we implement a FakeAction type using the Writer monad:

type FakeAction = Writer [String]

This is a monad which aggregates the log messages of type String. We specify what logInfo means in this context:

instance HasLogFunc FakeAction where
  logInfo msg = tell [msg]

Now we can express the always failing scenario:

describe "retry" $ do
  it "gives up after the specified time" $
    runWriter (retry 3 alwaysFails) `shouldBe`
    ( Nothing
    , [ "Retrying 1/3"
      , "Retrying 2/3"
      , "Retrying 3/3"
      ])

runWriter returns the result of the provided action, in this case Nothing, paired up with the log messages which were produced during the evaluation. FakeAction is pure, it doesn’t do any IO, but it helps us to test retry‘s behavior.

This testing strategy can be further extended to cover other types of effects. In the complete code samples on GitHub you’ll see how I implemented exponential back-off: in production retry waits some seconds before it re-executes the action, however the test code is pure, running fast without any side-effects.

Summary

In this post I presented a simplified version of this pull-request which adds a simple retry mechanism to cache-s3.

I showed that we can make a function more testable by making it a more generic. Polymorphism allows us to inject test points into our function. This is often achieved by mocks and fakes in traditional programming languages. In Haskell exploiting polymorphism is particularly elegant because we can code against powerful, abstract interfaces such as the monad.

The code is available on GitHub.

Consistent vocabulary in control flow

2019-09-25T00:00:00+02:00

Mainstream programming languages provide various constructs for control flow: conditionals, loops, exceptions, etc. Many of these can be modeled using the general functor concept. In this post I’m going to show you how.

In most programming languages defining a function (or a method, subroutine, procedure, etc.) is simple. Using this function in a real program is more complex: the function’s arguments may be missing (None, NULL, nil, etc.), the computation in the function may fail because of an unexpected condition, the function’s input argument may be the result of an asynchronous call.

In a real program a function call appears in a specific context with a particular control flow. In a given context specific edge cases appear which we must handle as well.

In the next sections I show you how a simple function is typically used in different contexts. I’m using examples written in Python and Haskell.

Modeling a camera

As a working example, let’s model a camera with a projection from a three-dimensional world point coordinates to two-dimensional image point coordinates.

In Haskell the type signature of such a function is:

project :: WorldPoint -> ImagePoint

where I assume that some reasonable definitions of WorldPoint and ImagePoint exist. That is, given a world point project returns a point on the camera’s image plane.

In Python the outline of this function would read:

def project(world_point):
   ...
   return image_point

The concrete implementation of project is not important. For the sake of this article we can assume that all relevant camera parameters are available in the function’s body.

Let’s see how we could use the project function in various contexts.

Simplest case

We designed our function to be pure, free of side-effects. Projecting a single world point is just a matter of calling project. It’s easy both in Python

project(world_point)

and Haskell:

project worldPoint

In a real world program, however, the situation is rarely that simple.

Missing value

Imagine that a preceding computation provides an empty world point to our program. We would like to use project in this context where the world point may not be present.

In Python, the missing value could be represented as a None value. Let’s use a conditional to check for it:

if world_point:
    image_point = project(world_point)
    # use image_point
else:
    # handle the missing value

This is a common pattern, you find such conditionals in every code base.

In Haskell we could accept the world point wrapped in a Maybe type and use fmap to compute a projection in case the world point value is present:

let maybeImagePoint = fmap project maybeWorldPoint

where the type of the input and output values are Maybe WorldPoint and Maybe ImagePoint, respectively.

This works, because Maybe is a functor. This means we can use the function fmap to lift our pure project function to operate on potentially missing values.

Handling a scene

We rarely work with a single world point, but with a collection of world points which I call here a scene.

If we wanted to project an entire scene on a camera, in Python we could use a list comprehension and write:

image_points = [project(world_point) for world_point in scene]

or more succinctly, using the built-in map function:

image_points = list(map(project, scene))

In Python 3, map returns an iterator which we explicitly convert to a list.

In Haskell, choosing a simple list representation for Scene, we write:

-- type Scene = [WorldPoint]
let imagePoints = fmap project scene

where the type of imagePoints is a list of image points.

This is almost identical to the Python code using map. List is a functor, fmap on list applies the provided function on each element. This is exactly map as we know it from Python.

Input from a data file

Let’s imagine that the world point is stored on disk in a data file. Before we can apply project we need to read and decode the data file.

In Python, we wrap the file operation in a try-except block:

try:
   world_point = read_data_file(...)
   image_point = project(world_point)
except (DecodeError e):
   # handle the error
   ...

This is also fairly common pattern: the exception handler, if we don’t forget to write it, allows us to handle the potential errors.

In Haskell, it’s again just fmap:

-- worldPointOrError :: Either WorldPoint DecodeError
-- imagePointOrError :: Either ImagePoint DecodeError
let imagePointOrError = fmap project worldPointOrError

Here Either WorldPoint is the functor and fmap acts as follows: if the decoding is successful project is applied on the returned value. In case of error the project function is not used at all and imagePointOrError will contain the returned error value.

Because the potential failure is encoded in the data type we cannot forget about error handling: that code would just not compile.

Asynchronous request

Finally let’s assume that we read the world point from the network. Network communication requires a lot of input/output so let’s do it concurrently with other tasks of our application.

In Python, using the asyncio library we can write concurrent code using the async/await syntax:

async def read_from_network():                  # ①
    await asyncio.sleep(1)
    return (0, 1, 2)  # dummy value

async def async_image_point():                  # ②
    return project(await read_from_network())

image_point = asyncio.run(async_image_point())  # ③

read_from_network is a coroutine simulating an asynchronous action obtaining the world point. In a real application this operation would run concurrently with other coroutines.
async_image_point applies the projection on the value returned by read_from_network. This is also a coroutine and it completes after the network communication has finished.
asyncio.run blocks until the provided coroutine delivers its return value. image_point is now a regular image point value.

Now let’s see how something similar works in Haskell:

readFromNetwork :: IO WorldPoint            -- ①
readFromNetwork = do
  threadDelay 1000000 -- µs
  return (0, 1, 2)    -- dummy value

imagePoint :: IO ImagePoint
imagePoint = do
  worldPoint <- async readFromNetwork      -- ②
  wait (fmap project worldPoint)           -- ③

readFromNetwork simulates the network IO: the body of this function can be replaced with calls to a real networking library which do not know anything about asynchronous operations.
Spawn the network operation asynchronously in a separate (lightweight) thread. Note that async is just a library function, not a special keyword.
We use fmap to lift the transformation into the asynchronous computation, then wait blocks until the asynchronous action completes.

By looking at the type signature of fmap specialized to this example:

fmap :: (WorldPoint -> ImagePoint) -> Async WorldPoint -> Async ImagePoint

We can see that fmap constructs a new asynchronous action where the provided function is applied to the result of the provided asynchronous action. It reaches under the Async constructor and transforms the underlying values.

Summary

We looked at how a pure function is used in four different computational contexts: missing values, collections, potentially failing and asynchronous actions.

In Python we used different, specialized language constructs to apply our project function:

Conditional: to handle the case when the input point may be missing
List comprehension (or loop): when the function is applied on a collection of points
Try-except block: when the data file decoding may fail
Special keywords async/await: when the function operates on results of asynchronous computations

In Haskell we always used fmap, because these seemingly different computations can all be modeled as a functor. Instead of using special language keywords we used one organizing principle borrowed from Mathematics where the control flow and the edge cases are precisely defined.

You can read more about functors on the Haskell wiki or elsewhere on the Internet.

For the topic I took inspiration from the talk The Human Side of Haskell by Josh Godsiff.

Concurrency without magic

2019-07-15T00:00:00+02:00

In a previous post I demonstrated some of Haskell’s features to write concurrent programs. I developed a simulated search engine which looked like this:

search30 :: SearchQuery -> IO ()
search30 query = do
    req <- timeout maxDelay $
        mapConcurrently (fastest query) [Web, Image, Video]
    printResults req

I argued that this was a fast, concurrent, replicated and robust version of the previous iterations. In the following sections I am going to improve this example, therefore I suggest reading the related post to understand how we got here.

The faster wins

In the heart of the search30 function we find this helper function:

fastest :: SearchQuery -> SearchKind ->  IO String
fastest query kind = do
    req <- race (fakeSearch query kind) -- server 1
                (fakeSearch query kind) -- server 2

    return $ case req of
        Left  r -> "Server1: " ++ r
        Right r -> "Server2: " ++ r

fastest sends the search query to two replicas of the search backend and keeps the result from the faster responding server. The race function from the async library runs two actions concurrently and returns the result of the faster. The slower action is terminated.

In real code you would replace the fakeSearch function with an appropriate search API call but the structure of the function would not change significantly.

For the sake of this post we explicitly prepend the name of the server from which the result came. In a real application this would not be displayed to the user. However, we could still store some information about the response’s origin for further analysis.

This implementation of fastest works well, but only in the special case of two servers. What if we want to use more replicas? Let’s see how we can add support for any number of back-end servers.

More general race

Instead of racing two processes, we start N instances of the fakeSearch action concurrently and receive the result of the fastest. This is a recurring pattern in concurrent programming: start many computations and signal if any of them completed execution. In many programming languages, for example Python and C#, you find library functions to solve this problem. The naming and the implementation slightly changes from one language to another, but the underlying concept is the same.

The race function served us well, so let’s keep digging in the async library for something that can help us out again. It doesn’t take too long to locate these two functions:

waitAny :: [Async a] -> IO (Async a, a)
-- Wait for any of the supplied Asyncs to complete.

waitAnyCancel :: [Async a] -> IO (Async a, a)
-- Like waitAny, but also cancels the other asynchronous
-- operations as soon as one has completed.

This looks promising: we can use waitAnyCancel to achieve the same behavior as race but for a list of asynchronous operations.

We cannot just replace race with waitAnyCancel because the type signatures of the two functions are different. For reference, this is race:

race :: IO a -> IO b -> IO (Either a b)

We need to think a bit more how to fit waitAnyCancel into fastest.

Building blocks

The central type of the async library, Async a, represents an asynchronous operation that yields a value of type a. Asynchronous operations can be spawned using the async function:

async :: IO a -> IO (Async a)

The type of the expression fakesearch query kind is IO String. Feeding this to async, async (fakesearch query kind), we get an IO (Async String) which launches the search asynchronously in a separate thread. We could replicate this N times and use waitAnyCancel to select the fastest query. However, if we went down this path we would be unable to identify which server replica gave us the fastest answer. We would only get back the search result, but not the winner replica’s identity.

When we used race it was easy to identify the faster replica because we used pattern matching on the returned Either value. waitAnyCancel does not give us any hint on which action in the provided list was the fastest.

Now, because IO a is a functor, we can use fmap (usually abbreviated as <$>) to apply a function on the result of the search operation. Let’s write a function which prepends the server’s identity to the search result:

-- prepend the server's identity to the search result
servedBy i result = "Server " ++ (show i) ++ ": " ++ result

For example, in case of the second server replica the transformed search action would read: servedBy 2 <$> fakeSearch query kind. The type of this expression is still IO String but it yields the search result with the server name prepended.

Let’s go over the series of transformation steps and see how the expressions and their types were modified:

-- the search API
fakesearch :: SearchQuery -> SearyKind -> IO String

-- fill in the arguments to get IO action that yields a string
fakesearch query kind :: IO String

-- prepend second replica's identity to the search result
servedBy 2 <$> fakesearch query kind :: IO String

-- a function which prepends the provided replica number to the search result
\i -> servedBy i <$> fakesearch query kind :: Int -> IO String

-- same as before but as an asyncronous action
\i -> async (servedBy i <$> fakesearch query kind) :: Int -> IO (Async String)

In the last two lambda-expressions I kept the replica number as a free parameter, a form we can reuse in the final step.

Final implementation

Let’s assemble the new fastest implementation from the pieces we have:

fastest :: SearchQuery -> SearchKind ->  IO String
fastest query kind = do
    requests <- forM [1..numReplicas] $ \i ->                           -- ①
        async (servedBy i <$> fakeSearch query kind)  -- <$> is `fmap`  -- ②
    (_, result) <- waitAnyCancel requests                               -- ③
    return result                                                       -- ④

  where numReplicas = 3
        servedBy i result = "Server " ++ show i ++ ": " ++ result

① Use forM to transform a list of integers, the replica identifiers, by applying the provided function. requests has a type [Async String], exactly what waitAnyCancel needs.

② The second argument of forM is the transformation function which creates an asynchronous operation yielding the search result with the replica’s identity prepended.

③ Call waitAnyCancel and extract the result from the fastest search operation (we don’t use first element of the returned tuple).

④ Provide result of the current IO operation.

This implementation, only six lines of code, works with any number of replicas. For numReplicas=2 its behavior is identical to that of the old one.

Summary

We replaced the original implementation of the fastest function, only capable of supporting two server replicas, into a more general one which works with any number of replicas.

The implementation relies on a single library function waitAnyCancel but we needed to arrange the asynchronous search operations in a way that is compatible with waitAnyCancel‘s type signature. We used generic combinators such as fmap and forM to achieve this. The search30 requires no modifications, as the type signature of fastest didn’t change.

We wrote concurrent code with no locks, no mutexes and no concurrent design patterns to remember. We manipulate asynchronous computations as values and call library functions. This is possible because the async library exposes generic and composable primitives and hides the complexity of thread management. Also the language provides powerful combinators such as fmap for implementing our programs.

You can find the examples given here as executable code on GitHub.

The inspiration of this and the previous post came from the Go examples presented in Rob Pike’s Go Concurrency Patterns talk.

Build systems à la carte

2019-06-19T00:00:00+02:00

Today I went to the talk “Build systems à la carte” of Andrey Mokhov. He presented the results of the identically named paper from last year authored by himself and his co-authors. The following links point to the paper and its related resources:

paper submitted to ICFP 2018
presentation from ICFP 2018 by Simon Peyton-Jones (co-author)
extended version of the original paper
slides from today’s presentation (the talk was not recorded)
code examples on GitHub

I had known about this paper from last year and I was excited to learn that Andrey was going to present this work at EPFL, Lausanne. In the next sections I summarize what I learned from the paper and the presentation.

Build systems

The study explores the design space of build systems such as Make, Ninja, Bazel. Interestingly, through the lens of this paper, systems like Excel and Docker can also be seen as build systems.

Typically we specify our builds as a set of rules: we tell how a given target is built out of its dependencies. The build system’s job is to bring a specified target up-to-date. Depending on the application domain a build system may have some of the following properties:

Minimality: don’t repeat work unnecessarily
Early cutoff: stop when nothing changes
Cloud builds: save repeating work by sharing build results among all developers
Dynamic dependencies: some dependencies are not known in advance, but they are discovered as we run the build.

Classification space

Two key design choices are typically deeply wired in any build system:

Scheduling: the order in which tasks are built
Rebuilding: whether or not a task is rebuilt

Today’s most commonly used build systems can be classified along these two axes. Let’s see these two aspects in detail.

Scheduling strategies

Build systems use different strategies to execute the build tasks:

Topological: all the dependencies are known in advance. The tasks are executed in topological order. Examples: Make, Ninja, Buck
Restarting: if a task has an non-built dependency its build is aborted and restarted later when the dependency is available. Examples: Bazel, Excel
Suspending: a task is suspended when its build encounters a missing dependency. Examples: Shake, Nix

The chosen scheduling strategy has an effect on the properties of the resulting build system:

A topological scheduler cannot support dynamic dependencies. A topological order can only be established if all the dependencies are known at the beginning of a build.
Build system using restarting schedulers are not minimal because some work is repeated when restarted. Note that the cost of duplicate work may often be just a fraction of the overall build cost.
A suspending scheduler is theoretically optimal, but it is only better in practice than a restarting scheduler if the cost of avoided duplicate work outweighs the cost of suspending tasks.

Rebuilding strategies

We find the following techniques to decide whether or not a task is rebuilt:

Dirty bit: anything that changed since the last build is marked dirty (Excel, Make)
Verifying traces: rebuild a target if the values/hashes of the its dependencies changed since the last build (Ninja, Shake)
Constructive traces: like verifying traces, but also store the resulting target value. The stored value can be shared with other users. (Bazel)
Deep constructive traces: like constructive traces, but only store terminal input keys ignoring any intermediate dependencies (Buck, Nix)

Again, picking a given rebuilding strategy has interesting consequences:

We can achieve minimality using dirty bits.
It’s possible but hard to support early cutoff with dirty bits. Make approximates early cut-off.
All traces support dynamic dependencies and minimality
All traces except for deep traces support the early cutoff optimization
Constructive traces enable cloud builds
Deep constructive traces cannot support early cutoff
Deep constructive traces may generate frankenbuilds if the tasks are not deterministic

Executable build system models

After a detailed classification of build system the authors present executable Haskell code to model real-world build systems. The concrete implementations are broken down into two components: Scheduler and Rebuilder, Here are some models in their final form:

make = topological modTimeRebuilder
excel = restarting dirtyBitRebuilder
shake = suspending vtRebuilder
bazel = restartingQ ctRebuilder

It’s remarkable that the concepts presented in this work lead such succint implementations. The actual executable code can be found in this repository.

Summary

The work “Build systems à la carte” from Andrey Mokhov and his co-workers taught me a lot about how build systems work. It demonstrates the power of Haskell as a modeling language. I recommend you to read the paper and study the related resources shown at the beginning of this article.

Exploring parser combinators

2019-05-03T00:00:00+02:00

This is an experience report of playing with Megaparsec, a parser combinator library in Haskell.

Parser combinators

The first time I saw a code snippet using the Parsec library I was truly amazed: a parser of Comma-Separated Values(CSV) reads:

csvFile = line `endBy` eol
line = cell `sepBy` (char ',')
cell = quotedCell <|> many (noneOf ",\n\r")

-- quotedCell = ... -- the definition is omitted

I had tried writing a CSV parser by hand before: it was about opening a file, looping over all the lines, splitting the lines on the comma, looping over those parts and so on. In the csvFile example I didn’t find any of those: I can see that the code describes the CSV file, but where is all the parsing happening?

In the previous code-block some notations can be unfamiliar, but this is a great example of declarative code: you specify what, and not the how. In fact, if you learn a bit the library, the code almost reads as plain English:

A CSV file is zero or more occurrences of lines, separated and ended by end-of-line
A line is zero or more occurrences of cells, separated by comma
A cell is either a quoted cell or zero or more characters, excluding comma and new-line.

Using this domain specific language results in code that is not only declarative, but also compositional: complex parsers are built out of simpler parsers using the provided combinators. In our example the line, cell and the quotedCell parsers can be developed, tested and maintained separately.

Developing parsers in this way is called combinatory parsing and has a rich literature.

If you ever played with the big guns of parsing such as Lex/Yacc or ANTLR you can appreciate a whole new level of expressiveness. The parser combinators are written as a library of the host language. Here, I am interested in Haskell, but Parsec-like libraries exist for other languages as well. Contrary to traditional parser generators, there’s no need for preprocessing or external tooling. In Haskell you can even try your parsers in an interactive REPL session. These features make it cheap to write ad-hoc parsers in your programs.

Haskell libraries

Many parser combinator libraries were written in Haskell; I did some research to decide which one to use. I narrowed down my options to three:

Parsec
Attoparsec
Megaparsec (fork of Parsec)

I read somewhere that the original Parsec library may not give the best error messages when a parser fails. attoparsec is designed to be super-fast, aimed particularly at dealing efficiently with network protocols and complicated text/binary file formats. Megaparsec promises a “nice balance between speed, flexibility, and quality of parse errors”. Sounds good to me. I decided to give Megaparsec a try.

Despite their different internals, these libraries expose a similar interface. Often it is not too difficult to port code from one library to an other. The README of megaparsec contains a more detailed comparison with other solutions.

Using Megaparsec

In spite of Megaparsec’s detailed documentation and its great tutorial, initially I had a hard time finding the relevant documentation for specific combinators. The Megaparsec library organizes its functions in multiple modules/packages:

parser-combinators: generic combinators such as some, optional and sepBy.
Megaparsec: running parsers, primitive and derived combinators. For example, parse, oneOf, noneOf
Megaparsec.Char: characters and character groups. For example, space, eol, tab, alphaNumChar, digitChar
Megaparsec.Lexer: high level parsers for handling comments, indentation and numbers. For example, skipBlockComment, decimal, signed

I think this structure is sensible, but it takes a bit to understand. Especially for beginners it is hard to grasp what is a primitive and derived combinator or what is considered as high-level and low-level parser.

Solving a typical parsing problem, such as parsing a CSV-file, requires only a handful of library functions. As soon as you have those combinators figured out the development becomes a breeze. I felt that Megaparsec lets me almost directly transcribe data format specifications into code.

Summary

Combinatory parsing is a great way of developing parsers. Parser-like libraries demonstrate many good aspects of library design. They are compositional but beginners may have a hard time initially dealing with abstract combinators.

If you want to learn more about this topic I recommend watching Scott Wlaschin’s presentations on parser combinators.

Load balancer

2019-01-30T00:00:00+01:00

In this post we are going to write a simple load balancer in Haskell. The design is based on that presented in Rob Pike’s Concurrency Is Not Parallelism talk (starting around 22 minutes). If you are not familiar with this presentation I highly recommend watching it before reading on. Pike presents an interesting load balancer design and its implementation to show off Go’s built-in concurrency primitives. Just like last year, I am interested how this example would look in Haskell, a purely functional programming language.

I am presenting some code fragments of the implementation omitting some less-important, plumbing details. You can find the complete executable code in this repository.

Design

A load balancer distributes workload across multiple computing units. Load balancing aims to optimize resource use, maximize throughput, minimize response time, and avoid overload of any single resource. I will call our “computing unit” simply Worker.

I have reproduced here the outline of the system we are going to implement:

                                   To requester  <──┐
┌───────────┐                                       │
│ Requester │────┐   ┌──────────┐       ┌────────┐  │
└───────────┘    │   │          │─────> │ Worker │──┘
                 │   │          │       └────────┘
┌───────────┐    └─> │          │       ┌────────┐
│ Requester │──────> │ Balancer │─────> │ Worker │────┐
└───────────┘    ┌─> │          │       └────────┘    │
                 │   │          │       ┌────────┐    │
┌───────────┐    │   │          │─────> │ Worker │──┐ │
│ Requester │────┘   └──────────┘       └────────┘  │ │
└───────────┘                                       │ │
                                   To requester  <──┘ │
                                                 <────┘

The design comprises three components:

Requester: Sends a unit of work to the system and waits for the result.
Balancer: Receives requests and distributes them among the available workers.
Worker: Executes the requested the work and sends back the result to the Requester.

The components communicate through channels. I shall use unbounded FIFO channels from the stm library.

The interesting part about this design is that the results from the workers do not cross the Balancer. The worker directly sends back the result to the requester.

Request

First, let’s define the data structure to hold the requests. This is a type of a request that, when executed, yields a value of type a:

data Request a = Request (IO a) (TChan a)

The Request holds an IO action and a result channel: the action is the executable task at hand, the result channel transmits the result back to the requester. Both the action and the channel are parametrized over the same type: the result of the task must “fit” in the result channel.

The task and the Requester

Let’s start imagining how the clients would interact with the load balancer. We generate some work with randomTask, a function with the signature:

randomTask :: IO Int

Neither the load balancer nor the worker have to know about the internals of this function. In my implementation randomTask draws a number from a uniform distribution, it sleeps that amount of seconds and returns the number as a result. In other words: the result of randomTask is the time it took to complete it.

The requester function represent the clients who have some work to perform:

requester :: TChan (Request Int) ^ -- input channel of the load balancer
          -> IO ()
requester balancer = forever $ do
  -- simulating random load
  delayMs <- getStdRandom $ randomR (1, 1000)
  threadDelay (delayMs * 1000)

  -- send the request
  resultChan <- newTChanIO
  atomically $ writeTChan balancer (Request randomTask resultChan)

  -- wait for the result
  async $ atomically $ readTChan resultChan

This function is an infinite loop where each iteration sends a request to the load balancer after a random delay. This is a very primitive way to simulate some load: not realistic, but good enough for now. The requester creates the result channel, packages it up along with the task, sends the request, and waits for the results asynchronously. The next iteration will not be blocked.

Note that the interface to the load balancer is a simple value. The requester only drops this value in a channel and waits. It does not care about what happens to the Request on the other end of the channel.

Balancer

As shown in the design, the Balancer receives the Request from a channel and distributes them among the available workers.

The Balancer holds a pool of workers and a completion channel:

data Balancer a = Balancer (Pool a) (TChan (Worker a))
-- Pool and Worker will be defined later

Workers use the completion channel to report to the Balancer the completion of a task. The Balancer uses this information to keep track of the load of each worker.

balance
  :: TChan (Request a) -- ^ input channel to receive work from
  -> Balancer a        -- ^ Balancer
  -> IO ()
balance requestChan (Balancer workers doneChannel) = race_ -- ❶
  runWorkers
  (runBalancer workers)
 where
  runWorkers = mapConcurrently
    (`work` doneChannel)
    (map snd $ toList workers)                             -- ❷
  runBalancer pool = do
    msg <-                                                 -- ❸
      atomically
      $        (WorkerDone <$> readTChan doneChannel)
      `orElse` (RequestReceived <$> readTChan requestChan)

    newPool <- case msg of                                 -- ❹
      RequestReceived request -> dispatch pool request
      WorkerDone      worker  -> return $ completed pool worker

    runBalancer newPool                                    -- ❺

As expected, this is the most complex part of the system. Let’s walk through it step by step:

The balance function runs the workers and the balancer itself asynchronously. The race_ combinator is from the async package and I talked about it in a previous post. It runs its two arguments concurrently and it terminates if any of those two terminate.
runWorkers concurrently executes the workers. The function work :: Worker a -> TChan (Worker a) -> IO () is part of the Worker API and makes the worker process requests in an infinite loop. I shall present the details of the Worker in the next section.
The balancer waits for new messages on the request and completion channels.
Depending on the received message runBalancer calls either dispatch or completed. These two functions implement the load balancing strategy and update the state of the worker pool.
runBalancer recursively calls itself with the updated worker pool.

The RequestReceived and the WorkerDone are data constructors of an internal type ControlMessage:

data ControlMessage a = RequestReceived (Request a)
                      | WorkerDone (Worker a)

This type “tags” the incoming message so we can differentiate from where it was originated.

The communication channels of the Balancer are now wired up; we are ready to implement the load balancing strategy. Following the the original design, I shall implement the worker pool using a heap from the Data.Heap module.

type Pool a = DH.MinPrioHeap Int (Worker a)

This heap stores priority-worker pairs (Int, Worker a). The priority represents the number of tasks assigned to the worker. Because we are using MinPrioHeap the pair with minimal priority, that corresponding to the least loaded worker, is extracted first.

The two functions dispatch and complete, introduced in the definition of balance, manipulate the heap of the worker pool. The function dispatch selects the least loaded worker and schedules the request on it.

dispatch :: Pool a -> Request a -> IO (Pool a)
dispatch pool request = do
  let ((p, w), pool') = fromJust $ DH.view pool  -- ❶
  schedule w request                             -- ❷
  return $ DH.insert (p + 1, w) pool'            -- ❸

The interface of the heap module is unusual, but the code is only three lines:

The function view extracts the worker with the minimum priority, that is with the lowest number of tasks.
The function schedule :: Worker a -> Request a -> IO () is part of the Worker API and it sends the given request to the extracted worker.
The function insert puts back the worker into the pool with an increased priority. The return value is the updated pool.

The completed function is very similar to dispatch:

completed :: Pool a -> Worker a -> Pool a
completed pool worker =
  let (p', pool') = DH.partition (\item -> snd item == worker) pool
      [(p, w)]    = toList p'
  in  DH.insert (p - 1, w) pool'

First, we look up the given worker using partition. Second, we put it back into the heap with a decreased priority value.

Worker

The last component of the system is the worker itself. The worker comprises an identifier, an integer in this case, and a channel from which it can receive requests:

data Worker a = Worker Int (TChan (Request a)) deriving Eq

This is the worker’s main function:

work :: Worker a -> TChan (Worker a) -> IO ()
work (Worker workerId requestChan) doneChannel = forever $ do
  Request task resultChan <- atomically $ readTChan requestChan -- ❶
  result                  <- task                               -- ❷
  atomically $ do
    writeTChan resultChan  result                               -- ❸
    writeTChan doneChannel (Worker workerId requestChan)        -- ❹

It is an infinite loop which (1) reads from the request channel, (2) performs the task, (3) sends the result to the requester, and (4) reports the completion to the balancer. We have already seen the implementations of other ends of these two channels.

For better encapsulation I provide a short helper function to send a request to the worker:

schedule :: Worker a -> Request a -> IO ()
schedule (Worker i c) request = atomically $ writeTChan c request

The function schedule deconstructs the Worker and sends the provided request to its internal channel. This function allows Worker to remain opaque and the request channel is not exposed outside the worker’s module.

Putting it all together

In the previous sections we have seen all the components: the requester, the balancer, and the workers. We only have to wire everything up:

start :: IO ()
start = do
  chan     <- newTChanIO          -- create a channel
  balancer <- newBalancer chan 3  -- create a `Balancer` with 3 workers

  -- run the balancer and the requester concurrently
  race_ (balance chan balancer) (requester chan)

If you run the complete code you will get an output like:

Balancer: fromList [(0,Worker 2),(1,Worker 3),(0,Worker 1)]
Balancer: fromList [(0,Worker 1),(1,Worker 2),(1,Worker 3)]
Balancer: fromList [(1,Worker 3),(1,Worker 1),(1,Worker 2)]

You can see that the first three requests were scheduled on different workers. The program continues indefinitely and you should see roughly the same amount of task on each worker.

Summary

I presented the Haskell implementation of an interesting load balancer design based on the Go code by Rob Pike. The load balancer gives the incoming request to the least-loaded worker. The components are loosely coupled and they communicate via channels.

Can we deploy this to production? Certainly not. The design seems to scale up to multiple workers and high request rates, but how can we prove this? What kinds of tests could we write for such a concurrent system? Can we test that the balancing strategy is sound? I will try to find answers to these questions in a future post.

The full code can be found in this repository.

Think paper

2019-01-03T00:00:00+01:00

Thinking using a computer is hard. First, a computer is full of distractions, but let’s say we can eliminate those. Second, using the conventional input such as mouse and a keyboard, anything more complex than a character or a mouse click is difficult to communicate to the computer. Where’s the button for my latest vague idea? Or where should I click to record an abstract concept?

As an experiment, let’s say you’re learning Math. Try recording digitally Pythagoras’s famous formula using the device you’re reading this post at:

$$ a^2 + b^2 = c^2 $$

Was it easy? In case you haven’t reached for your sleek tablet, opened your cloud-based note taking app which applies instant handwriting recognition and provides search functionality on Wolfram Alpha you may continue reading. But let’s say you type in the equation somehow. How do you attach a visual representation to the theorem? Maybe you can even jot down a simple proof? Can you represent your thought process as you’re working through all this?

Since at least 1968 human-machine interface designers want to increase the value created by an intellectual worker provided with a screen, a keyboard and a mouse. They actually did a pretty good job. Great digital tools help us to make beautifully typeset documents, videos, music and many more. In general, computers helps us enormously in doing, but I believe they inhibit us in thinking. Most of our thinking involve figurative or imagery terms and these representations of thought are very difficult to capture digitally. The way we interface with digital tools is simply too slow for doing fast iterations on a new vague idea that was just born in our head.

Imagine you are working on a chart for your next presentation. Take a pen and a piece of paper and in only a couple of minutes you can record five different versions your idea of that figure. You discard three because you can see immediately that they don’t communicate what you wish to express. In the next minute you realize that the remaining two versions could be combined into a completely new one you hadn’t considered before! In less then five minutes the sixth version of your graph must be something solid. Maybe not perfect, but it’s worth transforming your sketch into a digital figure. If you skip the initial paper sketches, how far do you get in PowerPoint in five minutes? I’m usually browsing on a support forum trying to figure out how to do a certain operation. If I’m lucky I get fancy bar chart in half an hour. I don’t like it, but I’m already out of ideas how to make it better.

When we find, for example, an influential talk on YouTube with meticulously prepared slides, what we see is the linear, condensed end result of a very personal, mostly non-linear thinking process. What we tend to forget is that creating is exactly that process and not the act of typing the next slide in PowerPoint.

I believe that in spite of the tremendous success of digitalization the thinking is still better done offline, with pen and paper. I find that these simple tools allow me to record and process my ideas the fastest. I cannot show off beautifully decorated Moleskin notebooks with charts and diagrams. I mainly scribble, draw boxes and potato-shaped thingies then I connect them with lines.

If you think for a living, think paper. Don’t stare at your big computer screens and try to come up with something smart by pressing some keys and clicking around. Get a piece of paper and record your thoughts with a pen. Draw, write, doodle. Give your ideas a structure on paper first then use digital tools after some initial iterations and reflection.

Minimum Coin Exchange

2018-04-30T00:00:00+02:00

In this post I solve the Minimum Coin Exchange problem programmatically using Haskell. I will compare the performance of the naive implementation to that using dynamic programming.

The problem

The minimum coin exchange problem is generally asked: if we want to make change for $N$ cents, and we have infinite supply of each of $S=\{S_{0},S_{2},\ldots ,S_{m-1}\}$ valued coins, what is the least amount of coins we need to make the change?

The solution can be found in a recursive manner where at each step of the recursion we have two options:

we use the $S_m$ valued coin in the change and we try to find the change for $N - S_m$ cents with the same set of coins.
we decide not to use the $S_m$ valued coin in the change: we keep on looking for a change of $N$ cents with $m-1$ coins

At each step we need to choose the option that uses less coins. It is expected that this process will end after finite number of steps because either the number of coins or the money to change decreases at each step.

On the algorithmist.com we find a succinct recursive formula to describe this process

$$ C(N,m)=\min {(C(N - S_{m}, m) + 1, C(N, m-1))} $$

the arguments of the $\min$ function correspond to the two options described above. The $+1$ in the first argument shows that choosing that option will increase the number of coins in the change. The recursive formula is completed with the base cases:

$$ \begin{align} C(N,m) &= 0, & N=0 \\ C(N,m) &= \infty, & N<0 \\ C(N,m) &= \infty, & N\geq 1, m\leq 0 \\ \end{align} $$

If the result of $C(N,m)$ is either $0$ or $\infty$ then it is impossible make change for $N$ with the given coins. The case (1) applies when we successfully changed $N$ with the given coins. Case (2) means that our smallest valued coin is larger than the amount for which the change is requested. Finally (3) represents the situation where we cannot pick any more coin to complete the change.

For simplicity, we will use a fixed set of coins: $[25, 20, 10, 5, 1]$. For example, the minimum number of coins needed for 40 cents using the formulation above is given by $C(40, 4)$. Also note that we only compute the number of coins, we don’t give the exact denominators to be used in the change.

Foundations

As we use a fixed set of coins we hard-code the available coin denominators:

-- Available coin denominators.
coins :: [Int]
coins = [25, 20, 10, 5, 1]

We will write a function with the following signature:

change :: Int     -- amount
       -> Int     -- index of the last available coin
       -> Change  -- the number of coins

change takes to arguments: the amount to change, the index of the last available coin and it returns the number of coins in the change.

Taking a list index as the second argument is, of course, very error prone: our program will crash if we try to address an element that is not present in the coins list. We ignore this deficency in order to stay as close as possible to the theoretical formulation of the problem.

Since there are cases where the change is impossible, we choose the result type Change as

type Change = Maybe Int

This is more expressive than using infinity as a sentinel value when the change is not possible.

With this preparation we are ready to implement the first version of change.

Naive implementation

-- Naive implementation using the recursive formula from:
-- http://www.algorithmist.com/index.php/Min-Coin_Change
change :: Int -> Int -> Change
change n m
  | n == 0            = Just 0
  | n < 0             = Nothing
  | n >= 1 && m <= 0  = Nothing
  | otherwise         = minOf left right

  where
    left = (+1) `fmap` change (n - sm) m
    right = change n (m - 1)
    sm = coins !! m

This implementation is almost maps one-to-one to the recursive formula above. The first three guard expression handle the three base cases. In the last clause the function calls itself to solve the two sub-problems which I called left and right.

Choosing Maybe to represent the result Change forces us to deviate from the clean mathematical formulation at two places:

we use fmap to add 1 to the result of the left subproblem
we use our own minOf function instead of the built-in min function

The first point is easy: we cannot add an Int to a Maybe Int because their types don’t match. Since Maybe Int is a functor so we can use fmap to lift the addition into the context of Maybe.

As for the second point, let’s see the implementation of minOf:

import Control.Applicative ((<|>))

minOf :: Change -> Change -> Change
minOf (Just i) (Just j) = Just (min i j)
minOf c1 c2             = c1 <|> c2

The function takes two Change values: if both Maybe contain Int values we choose the smaller one. Otherwise we try to keep the one that has a value in it using <|>. This behaves similarly to logical OR. We can easily test this in ghci:

Prelude> import Control.Applicative
Prelude Control.Applicative> Just 1 <|> Nothing  -- keeps the first
Just 1
Prelude Control.Applicative> Nothing <|> Just 2  -- keeps the second
Just 2
Prelude Control.Applicative> Nothing <|> Nothing -- returns Nothing
Nothing
Prelude Control.Applicative> Just 1 <|> Just 2   -- prefers the first
Just 1

This implementation of minOf gives us the right behavior when we select the smaller between the results of the left and right subproblems.

The built-in min can actually operate on Maybe Int values. We just cannot use it here because it returns Nothing if any of its argument is Nothing. This would terminate our recursive function without exploring the whole solution space.

We could almost directly implement the terse mathematical solution as a recursive function. Overall our function is short and readable, but before we open the champagne and celebrate let’s see how our solution performs.

Performance of the naive solution

We can use the microbenchmarking library criterion to measure the running time of the naive change implementation. Let’s see how the running time depends on the amount to change. The following table shows the approximate time of computing the change for 40, 100, 150 and 200 cents.

Amount [cents]	Running time [ms]
40	0.026
100	0.344
150	1.420
200	4.025

The running time of the naive solution scales polynomially with the number of cents. Let’s try to improve this!

Implementation using dynamic programming

The recursive method of the minimum coin exchange problem combines the solutions of subproblems with smaller amounts to change. We could optimize our naive solution using dynamic programming. The idea is that every time we solve a subproblem we memorize its solution. The next time the same subproblem occurs, instead of recomputing its solution we look up the previously solved solution.

We write a new function changeD which represents a stateful computation. The state is a map from problem parameters to its solution. In our case, a map from pair of integers (the denominator index and the amount) to a Change value. We call this computation Dyn:

import qualified Data.Map.Strict as M
import Control.Monad.Trans.State

type Dyn = State (M.Map (Int, Int) Change)

Using this, we can write changeD which, returns a Dyn computation resulting in a Change:

changeD :: Int -> Int -> Dyn Change
changeD n m
  | n == 0            = return $ Just 0
  | n < 0             = return Nothing
  | n >= 1 && m <= 0  = return Nothing
  | otherwise         = do

    left  <- memorize n (m - 1)
    right <- memorize (n - sm) m
    return $ minOf left (fmap (+1) right)

    where
      sm = coins !! m

The code looks very much like the naive solution but, since we’re operating in the Dyn context, we are using the do notation. The memorize computation implements the storing and recalling the subproblems’ solution:

memorize :: Int -> Int -> Dyn Change
memorize n m = do
    val <- do
        elem <- gets $ M.lookup (n, m)  -- try to recall the solution
        case elem of
            Just x  -> return x         -- return previously stored solution
            Nothing -> changeD n m      -- compute the solution

    modify $ M.insert (n, m) val        -- store the solution
    return val

This function is a literal implementation of the dynamic programming method. We try to look up the solution in the state: if the subproblem has already been solved we return the solution, otherwise we solve the subproblem and store its solution in the state.

The final step is to provide change in a form identical to the naive solution:

change :: Int -> Int -> Change
change m n = evalState (changeD m n) M.empty

We execute the Dyn computation using evalState by providing an initial empty state. This function provides exactly the same interface as the naive version above. The two implementations can be used interchangeably. Let’s which of the two implementations is worth using.

Naive vs dynamic

The table below compares the running times of the two implementations as a function of the amount to change.

Amount [cents]	Running time (naive) [ms]	Running time (dynamic) [ms]
40	0.026	0.117
100	0.344	0.354
150	1.420	0.556
200	4.025	0.814

The dynamic programming version scales linearly with the amount to change. The difference in running time becomes significant for amounts larger than 100 cents. As always, this speedup didn’t come for free: we traded running speed for storage space.

Summary

The minimum coin exchange problem is a classic example demonstrating dynamic programming. We implemented a solution by naively transcribing the proposed recursive formula almost literally to Haskell. We then used dynamic programming to improve time complexity of the naive solution. The dynamic programming method was encapsulated in a Dyn computation where the solutions to already solved subproblems are stored in a map.

The code for both implementations can be found here.

Knowing Algorithms

2018-03-10T00:00:00+01:00

The other day I got a question from a colleague: Do you know an algorithm for this problem of … details…details? The exact the problem description is not important. It was a well defined problem which totally made sense. I felt that there must be an algorithm for it, but I couldn’t help. I just simply didn’t know any.

I didn’t have a formal training on algorithms and complexity. Yes, I can easily write an $\mathcal{O}(n^2)$ sorting algorithm. Probably I can even make it $\mathcal{O}(n \log n)$, but I wouldn’t enjoy it. Of course, I quickly learned that I shouldn’t write my own sorting algorithm. Countless hours of thinking went into inventing, refining all sorts of algorithms so I better reach for libraries written by smart people and use those instead.

Most programming languages provide basic algorithms such as finding the minimum or maximum element in a sequence, sorting and so on. Sometimes we can get away with these “built-in” algorithms, but often we need to provide a more complex solution. In real world problems data are noisy, things can fail and just nothing is as simple as in textbook examples.

We may try to combine the provided basic algorithms as building blocks to solve our particular problem. Sounds reasonable, right? Tackle complexity by composition the mantra says. But if you’re given erase and remove_if to eliminate elements that fulfill a certain criterion from a C++ container, would you come up with the Erase-remove idiom? I wouldn’t. For example, in the talk STL Algorithms in Action from CppCon 2015 Michael VanLoon solves interesting sorting problems using components from the <algorithm> library. None of those solutions are trivial.

Building algorithms is hard, even if you have the basic components available. It takes special expertise and a lot of time. I don’t have this expertise, but I still want to help my colleagues to solve their domain specific problem.

The point Nicolas Ormrod makes in his great talk Fantastic Algorithms and Where To Find Them, is that algorithms are tools: you need to know they exist and you need to be able to use them. It doesn’t matter if you built the tool yourself (which is still very impressive), but how many different problems you can solve with it. The more tools you have in your toolbox, the more algorithms you know, the more problems you can solve.

One of piece of advice the book The Pragmatic Programmer gives (section Your Knowledge Portfolio in Chapter 1) is that you should learn at least one new programming language every year. I’d say you should also learn a new algorithm every month. So next time when you’re asked if you know an algorithm that solves a certain problem you can reach into your toolbox and answer: yeah, this algorithm can solve a problem which looks very similar to yours, maybe it would be a good fit…

Concurrency Patterns

2018-02-26T00:00:00+01:00

In this post I’m replicating in Haskell some of the examples from the talk Go Concurrency Patterns by Rob Pike. In the talk Pike explains how Go’s built-in concurrency primitives can help writing concurrent code. I was curious to see how the presented examples would look in Haskell, a language that I’m interested in learning.

Simulating a search engine

The example we are going to play with (again, taken from Pike’s presentation) is a simulated search engine. The search engine receives a search query and returns web, image and video results.

We start with a simple implementation where the different kinds of search are performed sequentially, then we gradually add concurrency to build a better performing search engine.

First we’re going to need some imports and some data types to represent our problem.

import Control.Concurrent.Async  (mapConcurrently, race)
import Control.Concurrent        (threadDelay)
import System.Random             (getStdRandom, randomR)
import System.Timeout            (timeout)
import Text.Printf               (printf)

type SearchQuery = String

data SearchKind
    = Image
    | Web
    | Video
    deriving (Show)

We represent the search query with a simple string and the kinds of search we can perform with product-type. The role of the imported functions will be clear as we go along.

This is already enough to get us started. We can write a fake search function which takes a SearchQuery, a SearchKind and returns an IO action which, when run, yields the search result as a string:

-- https://talks.golang.org/2012/concurrency.slide#43
fakeSearch :: SearchQuery -> SearchKind -> IO String
fakeSearch query kind = do
    delayMs <- getStdRandom $ randomR (1, 100)
    threadDelay $ microseconds delayMs -- simulating random work
    return $ printf "%s results for '%s' in %d ms" (show kind) query delayMs

    where
        microseconds = (* 1000)

Our search back-end won’t get any more sophisticated than this:

we make the current thread sleep for a random number amount of milliseconds, then
we return the simulated search results.

The resulting string contains the input parameters and the time it took to serve this request. We want to print these results so we write a small helper function:

-- Helper function to print the results
printResults :: Maybe [String] -> IO ()
printResults req = case req of
    Just res -> print res
    Nothing  -> putStrLn "timed out"

This function takes an optional list of strings and prints them on the console. If the argument is Nothing it means that the search request timed out (we will see this later). At this stage this function looks more generic than it needs to be. Probably you would start with a simpler signature, something like:

printResults :: String -> IO ()

and make it more complex when needed. This would make more sense and I also started with this initially. However, for the sake of this post, we will use the more generic version above so we can re-use it in the subsequent examples.

Search 1.0

The first version of our search engine will perform the image, web and video searches sequentially:

-- Run the Web, Image, and Video searches sequentally
-- https://talks.golang.org/2012/concurrency.slide#45
search10 :: SearchQuery -> IO ()
search10 query = do
    req <- mapM (fakeSearch query) [Web, Image, Video]
    printResults (Just req)

We use mapM to sequentially perform the three kinds of searches. The result of a typical search using haskell as a query would look like:

["Web results for 'haskell' in 27 ms",
 "Image results for 'haskell' in 61 ms",
 "Video results for 'haskell' in 17 ms"]

Since the searches are performed sequentially it takes 27 + 61 + 17 = 105 ms to serve the results to the user. This post won’t be about concurrency if we were to stop here.

Search 2.0

We can speed things up if we can send the three kinds of queries to our back-end servers independently and wait for the results to come back. We need to change one (!) function in our code to arrive to the next version of our search engine:

-- Run the Web, Image, and Video searches concurrently, and wait for all
-- results.
-- https://talks.golang.org/2012/concurrency.slide#47
search20 :: SearchQuery -> IO ()
search20 query = do
    req <- mapConcurrently (fakeSearch query) [Web, Image, Video]
    printResults (Just req)

The mapConcurrently combinator from the async library will launch three light-weight threads and perform the three search actions concurrently. The output of the program is similar (clearly the random timings will change), but it runs faster. The running time will be limited by the slowest search query.

We get a completely different behavior, but our code has the same structure as the sequential version. We don’t have to use locks, mutexes, callbacks, etc., the code clearly expresses our intent. It feels like we got concurrency for free.

Search 2.1

The concurrent version performs really well, but there might be cases where the slowest request would be very slow for some reason. Users get angry if things are slow, so we’d better display an error message if we cannot display a results within a given amount of time.

We introduce a maximum delay (80 ms in this example): if the slowest request takes longer then this, instead of the search results, we display a “timed out” message and we send our sincerest apologies to our user.

-- Don't wait for slow servers
-- https://talks.golang.org/2012/concurrency.slide#47
maxDelay :: Int
maxDelay = 80 * 1000 -- us

search21 :: SearchQuery -> IO ()
search21 query = do
    req <- timeout maxDelay $
        mapConcurrently (fakeSearch query) [Web, Image, Video]
    printResults req

Again, we only have to do a small modification to get the desired behavior. The search actions are wrapped in a timeout call. The signature of this function is:

timeout :: Int           -- maximum delay in microseconds
        -> IO a          -- the action to perform
        -> IO (Maybe a)  -- (Just a) if less than the delay elapsed else Nothing

We can run this version a couple of times and we will see two kinds of outputs. If all requests take less than 80ms printResults will behave as before:

["Web results for 'haskell' in 70 ms",
 "Web results for 'haskell' in 67 ms",
 "Video results for 'haskell' in 63 ms"]

otherwise, and this is why printResults takes a Maybe [String] as input,

timed out

is printed.

Search 3.0

We can still do better. The occasional timeout messages are still annoying. We won’t be too popular if our search website times out too often. It’s also a waste to discard all the results from slow servers. Maybe only the video search is slow, but we still throw away the web and image results.

We can use replication to reduce the chance of a timeout. We will send the request to two sets of back-end servers, two replicas. If one replica has some problem and responds slowly we can still get back a response from the other.

We write a function fastest that does exactly this:

-- https://talks.golang.org/2012/concurrency.slide#48
fastest :: SearchQuery -> SearchKind ->  IO String
fastest query kind = do
    req <- race (fakeSearch query kind) -- server 1
                (fakeSearch query kind) -- server 2

    return $ case req of
        Left  r -> "Server1: " ++ r
        Right r -> "Server2: " ++ r

This function have the same interface as fakesearch but internally it sends the same query two times to two different servers and keeps the result from the fastest. The race combinator from the async library:

race :: IO a -> IO b -> IO (Either a b)

helps us to achieve this. It launches the two IO actions in parallel and keeps the result from the fastest. The other action will be terminated. There is no second price in this race. The return value will indicate which action won. We prepend the server name to our search result to be able to see where it was served from.

We now replace the fakeSearch call with fastest and we get:

-- Send requests to multiple replicas and use the first response
-- https://talks.golang.org/2012/concurrency.slide#50
search30 :: SearchQuery -> IO ()
search30 query = do
    req <- timeout maxDelay $
        mapConcurrently (fastest query) [Web, Image, Video]
    printResults req

Let’s see some typical results:

$ search30
["Server1: Web results for 'haskell' in 29 ms",
 "Server1: Image results for 'haskell' in 36 ms",
 "Server2: Video results for 'haskell' in 49 ms"]

$ search30
["Server2: Web results for 'haskell' in 19 ms",
 "Server2: Image results for 'haskell' in 50 ms",
 "Server1: Video results for 'haskell' in 39 ms"]

We can see that in the first run the web and image results were served by the first replica, while the video result comes from the second. In the second example the first replica served the video results and the second replica the other two. Any combination is possible, these are just results from two runs.

With the version 3.0 we can serve the results of all three searches with very high probability within 80 ms.

Summary

As Rob Pike puts it, with a few transformations we converted a

slow
sequential
failure-sensitive

program into one that is

fast
concurrent
replicated
robust

I was surprised to see how little the code structure changed in the Haskell versions. We significantly changed the behavior of the program without compromising the readability. We didn’t have to use locks and mutexes, but we could focus on the intent, what the program should do, thanks to the powerful libraries and the run-time system.

I recommend to watch the original talk on YouTube and to further compare the Haskell and the Go implementations. You can find the code here.

Working habits

2016-07-20T00:00:00+02:00

Inarguably David Allen’s Getting Things Done had the great influence on me in developing an efficient working routine. I have implemented my own system based on the book which works well for me most of the times. Interestingly, GTD also taught me what I should not do, and when things go less great I am noticing it in a short time:

Maybe this should be a whole project rather than a task?
Is this item is really actionable?
Was there any progress on this project recently?
…

When these questions arise, it’s an indication that something needs to be changed. Actually, most of the time they signal that some thinking is needed before doing.

A key aspect of getting things done is a distraction free environment. You can have the best task system ever, if the phone always rings, message notifications pop up, projects will never move forward. I’d already recognized some points to improve in my working habits during my PhD, but I’ve been trying to shape my working environment more consciously since I started working in industry.

Here are some tools I use, techniques I implemented during the last years to support a good GTD work flow. Most of the my work happens before the computer, so some of the points following are rather specific.

So here are my tips (for myself and you):

Avoid distractions
Be asynchronous
Be responsive
Be distributed

Let’s see each point in detail.

Avoid distractions

As stated earlier this is the most important point. Distractions destroy productivity.

Solid black desktop

I used to tinker a lot with desktop widgets, I loved to put email notifications, clocks, cool backgrounds, nice window decorations, etc. on my desktop. Now they are all gone, they’re good for nothing but distractions. My desktop has solid black color, no icons or anything at all.

Tiling is great

I’m using a tiling window manager (XMonad) which helps me to keep my applications organized. I can only see the windows I need for what I’m working on. My task bar is very small and only shows the most essential information.

At most two screens

These days it’s common to have a workstation with two, three or even more screens. I used to love screens a lot myself: the more the better, I thought. It turns out that more screens just provide more space for distractions.

On this screen I will always keep my mail client open.

I used to think. Well, that’s basically the same thing as having an annoying “You’ve got mail” notification on the desktop. Very annoying.

At work, my workstation has two screens, which is sufficient for me. Two screen provides reasonable arrangements for most of my work flows. For example I can keep:

code and documentation,
code and testing/debugging outputs,
browser and mail client

comfortably next to each other. At home I use my laptop without external screens. Thanks to my window manager I can switch almost instantaneously between virtual desktops, so I don’t feel limited by its relatively small, 12.5” screen.

Be asynchronous

Most of these ideas I learned from a series of blog posts explaining How GitHub Works.

Prefer email to phone

I prefer email to phone. Receiving an email is not a distraction for me (see my comment on email notifications above) while a phone call is. I can read and answer emails when I want, at the time and location where I feel like dealing with them.

Of course there are good reasons to call: the feedback is immediate, the discussion is more personal, conveying a message may be easier. Still, most people prefer to call because they fear that the other person might not read their emails or forget to answer. Indeed, most people haven’t got the slightest clue how to deal with their email. The result is: emails get lost and forgotten so people think Maybe I should just call instead…

Calls and email can co-exist very naturally: calls can be organized over email. Fix a time then discuss what’s needed. In fact, this way the phone call effectively turns into a meeting. More on meetings in the next section.

Also email is only good as a communication tool if emails get answered, that is if the use of email comes with certain responsiveness. I’ll return to this point too later.

Prefer documentation to meetings

Just Google the terms “why meetings are bad” or “how to have good meetings” and you’ll see my point. The bottom line is that meetings are super expensive and very inefficient.

When I need to distribute knowledge I prefer to write documentation. No, not that kind of documentation everybody hates writing. I write a couple of lines on our company wiki or issue tracker, send a link around and ask people to read and ask questions or, even better, directly edit it if something is not clear.

Short essays, structured documents such as RFCs or PEPs are great to convey a proposal or an idea. These documents can be distributed, discussed and enhanced over email and no boring meetings are required. They don’t have to be super formal to be useful: the time spent on just gathering the words together helps to get out something solid out of the thinking process.

Be responsive

Asynchronous communication as mentioned before does not work without trust. People will bother you on the phone (or through other synchronous channels) if they fear that you won’t take the time to read your e-mails or review the documents you received. Trust can be build by being responsive. If people develop trust in me and learn that they will get a response from me in time, it helps them to adapt their work flows to mine.

I try to answer my e-mails in a ‘reasonable’ time. It depends on the context what this means:

work related mails I try to answer within a day.
a mail from a friend a subject ‘how is it going’ will be answered in a couple of days.

It happens that an email stays unanswered for a longer period of time, but most of the time it’s because I didn’t take the time to do it (which effectively means, because I was lazy) not because it was lost or forgotten. I use the ‘Inbox Zero’ strategy (the term coined by Merlin Mann) to handle my mails using a super simple system named ‘Trusted Trio’ adopted from Lifehacker.

Be distributed

Stuff gets done at physically different locations. For me these locations are: work, home and when I’m on the go. For example, it can happen that in the office during the day I take some notes that I need in the evening at home to complete a certain task. This means that my notes need to be distributed among all my places of work and they need to be accessible without too much effort. In GTD terms: my reference material needs to be accessible from different contexts.

More specifically:

It’s a commonplace, but I can access my e-mails from anywhere.
I synchronize my browser bookmarks and history using Firefox, so I can save interesting sites for reading them later, somewhere else.
My configuration files are stored on GitHub so I can access them from all my work stations.
My notes are in a text file in a Dropbox folder.
My task list is kept in sync by Taskwarrior.

I’d like to improve on my current setup to make my personal data, such as pictures, documents, etc. more accessible when I’m not home (only for myself in a secure manner of course). Maybe I write a post about this some other time.

Summary

I try to shape my working habits to get my stuff done in the most efficient manner. I identified four principles (no distractions, asynchronous communication, responsiveness and distribution) which can help me to achieve this.

Going mobile

2015-08-20T00:00:00+02:00

This is an experimental post, sort of proof of principle, to see how comfortable it is to write blog posts from my mobile phone. Maybe I should have called it hello world from Android.

Ingredients

Git based workflow I write the posts and generate the blog on my laptop (using GitHub pages).
Git client on my phone. After a bit of research I quickly settled with PocketGit. It is not for free, but costs only 1.79 CHF and, according to the reviews, this is the best Android client you can get.
An editor. Since on the desktop I use vim for everything I gave Vim Touch a shot, but was not convinced. For now I installed the Quoda code editor. So far so good, we will see how I like it.

Setup

There was really nothing unusual to do. Only PocketGit only needed some additional configuration. I generated an SSH key pair to be used by my phone to access the remote Git repositories. The rest of the infrastructure was already in place.

Conclusion

This first post entirely written on my Android phone is about to finish. It took me only about an hour to configure everything and I am overall happy with this tool chain.

Typing on my small phone is a bit painful but I think I can get use to it (otherwise I have to buy a phablet or suchlike). I will keep publishing the posts only from my laptop where I can comfortably spell check, polish and integrate the changes I made into the master branch. Also my blog generator does not run on Android, but this is not an issue at all.

Code archeology

2015-03-25T00:00:00+01:00

Back in 2004 a friend of mine was building a telescope and I wanted it to have an automated control system, where the user can select a star she wants to see and the system would automatically orient the telescope towards the appropriate celestial coordinates. I called this ambitious plan: “Project Starfinder”. The plan was that he builds the telescope and I write the software. We both did our job separately, but we never got around to put our works together.

Though Project Starfinder failed, the motor controller I built was quite a success. The other day I was browsing through my old backups and I stumbled upon the code I wrote 11 years ago. motorc and motord were the client and the server component, respectively, of my stepper motor controller.

Daemon

The motord daemon was responsible to receive messages from the connected clients and translate them to instructions to be written to the parallel port. I also built a circuit that could drive four motors connected to the PCs parallel port. The daemon needed to run with root privileges.

The daemon was accepting connections on a PF_LOCAL socket. For each incoming connection the daemon forked off a child process to handle the one to one communication with the connected client. I had read somewhere that Apache worked this way and I remember being quiet very to learn about fork().

I used select on an fd_set to process the incoming messages. This caused me a lot of headaches. I didn’t really understand what was going on with these file descriptors. Everything about them seemed magical. I probably copied the respective code from a tutorial or somewhere.

Anyhow, the messages arrived from the socket and they were processed in a large switch-case:

pid = fork (); /* forking off */

if (pid == 0) /* if we are a child process */
{
   select (FD_SETSIZE, &set, NULL, NULL, &timeout);
   while (1)
   {
      read (client, data, sizeof(t_motor_command));
      if (debug)
         syslog (LOG_DAEMON | LOG_DEBUG,
                 "Instruction: 0x%x, Address: 0x%x",
                 data->inst,
                 data->addr);

      if (data->inst == BYE)
         break; /* TODO: a safer solution needed */

      switch (data->inst)
      {
         case MOTOREN: /* Motor enable */
             motors[data->addr]->enabled = 1;
             break;
         case MOTORDIS: /* Motor disable */
             motors[data->addr]->enabled = 0;
             break;
         case MOTORR: /* Direction right */
             motors[data->addr]->direction = 1;
             break;
         case MOTORL: /* Direction left */
             motors[data->addr]->direction = 0;
             break;
         case MOTORCLR:     /* clear the position */
             motors[data->addr]->position = 0;
             break;
         /* ... */
       }
    }
}

This is the code from the source file almost literally. I used 8 spaces for indentation and apparently I didn’t care about lines longer than 80 characters back then.

I developed a simple binary protocol to communicate over the socket. The commands were represented by t_motor_command data structure:

typedef struct
{
     WORD inst;  /* instruction byte*/
     WORD addr;  /* motor address   */
} t_motor_command;

Pretty standard stuff.

The motor daemon also had the following features:

read and parsed a configuration file
supported arbitrary numbers of motors. In fact the motor objects were stored in a linked list.
wrote log messages to the syslog
changes in the motor position were made available to all clients
handled SIGTERM and SIGCHILD signals and made a clean exit
it was GPL licensed

I didn’t know how to use valgrind or gdb back then, so probably it leaked memory :-).

Graphical client

My memories about development of the client application are more vague: it was much faster to develop it, and also it is much simpler program than the daemon. I used Glade to design a GTK interface (with GTK 1.2). I tried to compile the project in order to make some screenshots of the interface, but I failed. The project depends on old libraries, moreover the newer version of Glade is having troubles reading the old interface files.

Anyway the interface was quite simple: it had four knobs for the four motors. As the user turned the knobs the motors turned. Very exciting! If you opened multiple clients the knobs were synchronized (there was no position feedback from the motors).

Summary

Reading through the code brought back some nice memories about this project. I had great fun designing the client-server architecture and I was really proud that I could handle multiple clients by forking off the server process. It’s a pity though that the code didn’t move any telescopes…

The ultimate eclipsometer™

2015-03-20T00:00:00+01:00

On the day of the solar eclipse I improvized a camera obscura in the office. It’s not nice, but functional and took only 10 minutes to make it. It was great fun to see my colleagues tucking their heads in the box to see the phenomenon.

I got a cardboard box from the store. I used a piece of plastic (cut out from a 3.5” floppy disk) to create a thin slit. The slit is fixed over a hole I cut in the side of the cardboard. An office chair provides the support.

I glued a white sheet of paper in the box, on the side that is opposite to the slit. The image of the eclipse is small, but surprisingly sharp.

My blogging workflow with git

2013-08-09T00:00:00+02:00

This is a description of my git workflow for writing blog posts. I will use this very post as an example and explain in detail how it was made.

Start a new topical branch

I create a topical branch named workflow (because I’m working on a post about my workflow) from the master branch

$ git checkout -b workflow

I edit the file content/2013-08-09-Git-workflow.rst and start writing the post. When I have enough typing for the day I commit the changes:

$ git add content/tech/2013-08-09-Git-workflow.rst
$ git commit -m 'Add post: Git workflow.'

It rarely happens that I finish a post in one go. When I feel like writing more I make some modifications (still on the workflow branch) and commit them:

$ git commit -a -m 'Continue workflow post.'

It typically takes two or three iterations, adding parts, fixing typos, etc. until the post is finished. In the end the topical branch contains quite a few commits since it was forked off from master:

$ git log master..
[there may be some more commits here]

commit 4f721af41b2c0d52a8456bb6735ad0f56754ea98
Author: David Wagner
Date:   Fri Aug 9 10:44:58 2013 +0200

    git log output.

commit f77de095d0d5225a566782c52133be7f65166016
Author: David Wagner
Date:   Fri Aug 9 10:44:30 2013 +0200

    Add summary.

commit e332a4b53c8cdd94f3117a949a99367b56a8bc6e
Author: David Wagner
Date:   Fri Aug 9 10:39:38 2013 +0200

    Add post: Git workflow.

Rebase on master and squash commits

I want to “compress” all these into one single commit, since there’s no point having commits in the master branch with messages such as “Add summary”, “Fix typo.”, “Continue this and that.”. In git’s words this is called interactive rebasing.

$ git rebase -i master

This pops up $EDITOR with a file to edit the rebase plan:

pick e332a4b Add post: Git workflow.
pick f77de09 Add summary.
pick 4f721af git log output.

# Rebase 6ed71cf..9be73d2 onto 6ed71cf
#
# Commands:
#  p, pick = use commit
#  r, reword = use commit, but edit the commit message
#  e, edit = use commit, but stop for amending
#  s, squash = use commit, but meld into previous commit
#  f, fixup = like "squash", but discard this commit's log message
#  x, exec = run command (the rest of the line) using shell
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#

The comments explain very clearly what I need to do: I modify the file to contain the following (omitting the comments):

pick e332a4b Add post: Git workflow.
squash f77de09 Add summary.
squash 4f721af git log output.

When the file is saved, I exit from the editor. Git starts the rebase offering to edit the commit message for each posts to remain, then finally reporting:

[...]
Successfully rebased and updated refs/heads/workflow.

Merge into master and publish

Now I switch back to master and merge, then delete the topical branch:

$ git checkout master
$ git merge workflow
$ git branch -d workflow

The new post is ready, the blog can be regenerated and published.

Summary

In short, I’m quite pleased with this setup. Using the combination of git and the Pelican static blog generator is really easy and has a lot of advantages: I can work on multiple posts at the same time, even offline and publishing is just a matter of a git push. The above workflow rose naturally, when I became to know enough about git’s branches and rebasing.

Evidently this whole thing may appear way more complicated than necessary for the faint hearted on Blogger, however this branch juggling turns out to be quite powerful when it’s about software development in the wild where clean commits and keeping track of changes you make on the codebase are essential.

Total control

2013-07-22T00:00:00+02:00

I ordered some electronic components from Play-Zone which made me really easy to control my unipolar stepper motor and read data from a 10k potentiometer.

Stepper motor

After sufficient preparation, it took me less than 15 minutes to have a nicely purring and turning stepper motor on the breadboard. The key element, a ULN2803A containing 8 Darlington Arrays, takes care of everything.

Potentiometer

Not strictly related, but in the same go I used the newly arrived MCP3002 ADC to read the position of a 10k potentiometer. I borrowed some code from the Adafruit website with some modifications from here.

#!/usr/bin/env python
import time
import os
import RPi.GPIO as GPIO

GPIO.setmode(GPIO.BOARD)

# read SPI data from MCP3002 chip, 2 possible adc's (0 thru 1)
def readadc(adcnum, clockpin, mosipin, misopin, cspin):
        if ((adcnum > 1) or (adcnum < 0)):
                return -1
        GPIO.output(cspin, True)

        GPIO.output(clockpin, False)  # start clock low
        GPIO.output(cspin, False)     # bring CS low

        if adcnum ==0:
            commandout = 0x6
        else:
            commandout = 0x7
        commandout <<= 5    # we only need to send 3 bits here
        for i in range(3):
                if (commandout & 0x80):
                        GPIO.output(mosipin, True)
                else:
                        GPIO.output(mosipin, False)
                commandout <<= 1
                GPIO.output(clockpin, True)
                GPIO.output(clockpin, False)

        adcout = 0
        # read in one empty bit, one null bit and 10 ADC bits
        for i in range(12):
                GPIO.output(clockpin, True)
                GPIO.output(clockpin, False)
                adcout <<= 1
                if (GPIO.input(misopin)):
                        adcout |= 0x1

        GPIO.output(cspin, True)

        adcout >>= 1       # first bit is 'null' so drop it
        return adcout

SPICS = 18
SPIMOSI = 19
SPIMISO = 21
SPICLK = 23

# set up the SPI interface pins
GPIO.setup(SPIMOSI, GPIO.OUT)
GPIO.setup(SPIMISO, GPIO.IN)
GPIO.setup(SPICLK, GPIO.OUT)
GPIO.setup(SPICS, GPIO.OUT)

# 10k trim pot connected to adc #0
potentiometer_adc = 0;

last_read = 0       # this keeps track of the last potentiometer value
tolerance = 5       # to keep from being jittery we'll only change
                    # volume when the pot has moved more than 5 'counts'

while True:
        # we'll assume that the pot didn't move
        trim_pot_changed = False

        # read the analog pin
        trim_pot = readadc(potentiometer_adc, SPICLK, SPIMOSI, SPIMISO, SPICS)
        # how much has it changed since the last read?
        pot_adjust = abs(trim_pot - last_read)

        if (pot_adjust > tolerance):
               trim_pot_changed = True

        if trim_pot_changed:
                # convert 10bit adc0 (0-1024) trim pot read into
                # 0-100 volume level
                set_volume = trim_pot / 10.24
                set_volume = round(set_volume)  # round out decimal value
                set_volume = int(set_volume)  # cast volume as integer

                print 'volume = {volume}%' .format(volume = set_volume)

                # save the potentiometer reading for the next loop
                last_read = trim_pot

        # hang out and do nothing for a half second
        time.sleep(0.5)

These two (independent) circuits make a big mess on the breadboard, but it was a lot of fun to wire it up. Now I can read analog as well as digital inputs from the real world with the Raspberry Pi! Yay!

Prototypes for stepper motor control

2013-07-19T00:00:00+02:00

I have a unipolar stepper motor I want to control with my Raspberry Pi. First I needed to figure out the motor coils’ wiring, then I wrote a small test script to provide the correct sequence on the GPIO ports.

Wiring

I read about stepper motors, and made the following circuit on the breadboard.

A 9V battery, an indicator led to show if the power is on, and four switches. When a switch is pressed one phase (half coil) is energized. For some unknown reason I managed to get the wiring right for the first time, so pressing the switches one after the other, from top to bottom, made the motor turn one step.

The 6 wires of this motor (Japan Servo KP68P2-406, 12V, 33 Ω/phase, 1.8 deg/step) are the following:

2 red (center taps)

orange (phase 1)

brown (phase 2)

yellow (phase 3)

black (phase 4)

Sequence test

I wrote a small test script to output the step sequence on the Pi’s GPIO ports. It has a lot of cool features:

$ sudo python stepper_seq.py -h  # help
$ sudo python stepper_seq.py  # turns the motor indefinitely, Control-C terminates
$ sudo python stepper_seq.py -s half -r -n 4  # 4 times the half-step sequence, reversed
$ sudo python stepper_seq.py -s kitt  # nothing to do with stepper motors ;)

And the code of stepper_seq.py:

#!/usr/bin/env python
# stepper_seq.py - Stepper motor sequence test
# Copyright (C) 2013 David Wagner
from itertools import chain, cycle, repeat
import argparse
import RPi.GPIO as GPIO
import signal
import sys
import time

sequences = {# Wave Drive, One-Phase
             'one': [[True, False, False, False],
                     [False, True, False, False],
                     [False, False, True, False],
                     [False, False, False, True]],
             # Hi-Torque, Two-Phase
             'two': [[True, True, False, False],
                     [False, True, True, False],
                     [False, False, True, True],
                     [True, False, False, True]],
             # Half-Step
             'half': [[True, False, False, False],
                      [True, True, False, False],
                      [False, True, False, False],
                      [False, True, True, False],
                      [False, False, True, False],
                      [False, False, True, True],
                      [False, False, False, True],
                      [True, False, False, True]]}

sequences.update({'kitt': (sequences['one'][:-1] +
                           list(reversed(sequences['one'])))})

parser = argparse.ArgumentParser(description='Stepper motor sequence test.')
parser.add_argument('-p', '--pins', metavar='num', type=int, nargs=4,
                    default=[11, 12, 13, 15],
                    help='GPIO pins to activate (numbering by BOARD)')
parser.add_argument('-s', '--seq', type=str, default='one',
                    choices=sorted(sequences.keys()),
                    help='Coil activation sequence.')
parser.add_argument('-d', '--delay', type=float, default='0.5',
                    help='Delay in seconds.')
parser.add_argument('-r', '--reverse', action='store_true',
                    help='Reverse direction.')
parser.add_argument('-n', '--num', type=int, default=0,
                    help='Number of complete cycles.')

args = parser.parse_args()


def init_board():
    GPIO.setmode(GPIO.BOARD)
    for pin in args.pins:
        GPIO.setup(pin, GPIO.OUT)


def init_pins():
    for pin in args.pins:
        GPIO.output(pin, False)


def signal_handler(signal, frame):
    print "Interrupted. Exiting..."
    init_pins()
    sys.exit(0)


def ncycles(iterable, n):
    "Returns the sequence elements n times"
    return chain.from_iterable(repeat(tuple(iterable), n))


def step(num=0):
    pin_order = list(reversed(args.pins)) if args.reverse else args.pins
    rep = cycle if num == 0 else lambda it: ncycles(it, num)
    for seq in rep(sequences[args.seq]):
        for pin, status in zip(pin_order, seq):
            GPIO.output(pin, status)
        time.sleep(args.delay)


def main():
    signal.signal(signal.SIGINT, signal_handler)
    init_board()
    init_pins()
    step(args.num)


if __name__ == '__main__':
    main()

Sequence test hardware

I put together a circuit to test the script above. It contains four LEDs, each connected to one of the Pi’s GPIO ports through a transistor. The LED’s could be directly connected to the IO ports, but in this project I wanted to refresh how to use a transistor as a switch so that the LEDs can be powered from an external power supply.

And on the breadboard it looks like this:

Led and button

2013-06-26T00:00:00+02:00

I finally got around to tinkering with the Raspberry Pi hardware. A while ago I’d ordered a basic electronic kit from ebay with a breadboard and a bunch of electronic components (resistors, cables, etc) in it. I started with a rather elementary project: control of an LED and a push-button from the Pi. Despite its apparent simplicity it’s a good opportunity re-learn some basics: which is the anode/cathode of the LED? What’s a pull-up/pull-down resistor? and so on…

Here’s the final ‘product’:

I needed to cut open up my fancy cardboard case :-( to gain access to the GPIO ports on the Pi. I may design a new case in the future, one that allows direct connection to the board.

I drew the circuit diagram using the circuit symbol set for Inkscape.

The drive this wonderful piece of hardware, I wrote the following code that makes the LED blinking (yaaaay):

# led_blink.py
import time
import RPi.GPIO as GPIO

# Set the mode of numbering the pins.
GPIO.setmode(GPIO.BOARD)
GPIO.setup(12, GPIO.OUT)

# Initialise GPIO10 to high (true) so that the LED is off.
status = True
while 1:
        GPIO.output(12, status)
        status = not status
        time.sleep(1)

and this program turns the LED on when the button is pressed:

# led_button.py
import RPi.GPIO as GPIO

# Set the mode of numbering the pins.
GPIO.setmode(GPIO.BOARD)
GPIO.setup(11, GPIO.IN)
GPIO.setup(12, GPIO.OUT)

# Initialise to high (true) so that the LED is off.
GPIO.output(12, True)

button_pressed = False
while 1:
    if GPIO.input(11):
        if not button_pressed:
                print 'Button pressed.'
                button_pressed = True
        GPIO.output(12, False)
    else:
        if button_pressed:
                print 'Button released.'
                button_pressed = False
        # When the button switch is not pressed, turn off the LED.
        GPIO.output(12, True)

I took the inspiration and some sample code from the following places:

Fresh theme

2013-06-18T00:00:00+02:00

I changed the blog’s theme from the default to fresh. I wanted to have something that uses Twitter Bootstrap so that if you resize the browser window, or read the posts on a small screen, the layout changes dynamically.

Also, now there’s a Github ribbon in the top right corner, and social links on the right.

Backup

2013-05-15T00:00:00+02:00

At work, I stored all my important data on the lab’s server which (obviously) was backed up once a day. At home, on my personal laptop I haven’t had a systematic backup solution. Now I’m changing this.

Everyday backup with Obnam

I decided that I will backup my laptop’s important data on my home file-server. It’s a DNS-320 ShareCenter with 2x1TB disk in RAID 1. Naturally, this won’t save me in case of theft or fire in my house, but I will deal with the off-site backup problem later, when a local backup problem is solved.

For my backups I use Obnam. My configuration file ~/.obnam.conf is qute simple:

[config]
repository = /media/nas/backup
log = /home/dwagner/.obnam/obnam.log
exclude = /home/dwagner/temp, /home/dwagner/pictures

To backup I run:

$ obnam backup $HOME

which takes a couple of minutes. Occasionally I make obnam forget the old files:

$ obnam forget --keep=14d

Of course I could put this in a little script and use cron or something to run the backup regularly, but I do it manually for a while to gain some experience.

I exclude the temporary files in my home directory because it just contains all kinds of junk I don’t really care loosing. I also don’t backup this way the pictures directory, since I secure my photos in there differently.

Photo backup with git-annex

I store my personal photo collection in a git-annex repository. I have at least two copies of all the pictures (one on the file server and one on a USB HDD dedicated for this purpose). A copy of the recent photos, usually the current year’s are on my laptop as well. I noticed that these are the pictures I access the most frequently, mainly showing them to friends and family. If I need to go back earlier in time, I can have those photos from home in a day (or when I go home), but not instantly from my laptop. This works for me, because the SSD on my laptop is not that spacious, and I don’t want to store all my photos on it.

When I download new pictures from my camera I do git-annex sync across the picture repositories making sure that my precious memories are well preserved.

Cardboard Raspberry Pi case

2013-03-17T00:00:00+01:00

I bought a Raspberry Pi. No need to explain this since nowadays everybody buys a Raspberry Pi. Even if I don’t use it, large part of its (not too high) price goes for charity, I figured.

I was reluctant to buy a casing for it in the shop, because I wanted to build it myself. There are tons of interesting RPi case projects out there, but I found Jude Pullen‘s the most interesting. I followed his instructions and the result is a very cool Raspberry Pi case out of cardboard. In fact, I recycled the packaging of my new ThinkPad.

I didn’t do the light-pipes. I’m not sure if this is the Pi’s final casing and I didn’t want to glue anything onto the board itself. In summary, working with cardboard is easy, fun and kept me away from the computer for some nights. I may build a better version (with light pipes), or even something completely different in the future.

Brain transplant

2013-03-12T00:00:00+01:00

In the lab we have a camera system composed of four cameras. The system is controlled by two identical PCs, two cameras connected to each. After a long break of operation, when we wanted to use the system again, the hard drive in one PC decided to quit science and went dead. We replaced the poor fellow with a new drive and I wanted to transfer the system as it is from the healty computer to the new drive of the braindead system . Following the brain transplant procedure in its most realistic, bloody purity.

My idea is to run on healthy:

dd if=/dev/hda | ssh user@braindead sudo dd of=/dev/sda

To this end I boot up braindead (with the new drive in) from a Debian Live CD. While setting up the network (IP address, DNS, etc.), it turns out that the network card needs some non-free firmware which are not on the Installer/Live CD. This is a bit annoying but not the end of the world. After some googling I learn that I need fimrware-linux-nonfree, so I download and put it on a USB stick and install (dpkg -i) it on braindead. Now it has network connection and an SSH server running.

To prepare for the transplant I put healthy in single user mode and mount the file system read-only (as root):

$ init 1
$ mount / -o remount,ro

Now everything is ready for the procedure. I run dd through SSH as described above. It takes some time, but the transfer works fine.

On braindead, still running the Live CD, I mount the new disk and

change the host name in /etc/hostname
generate new host keys (ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key)

I reboot braindead and voilá, it works! Well almost… I experience some problem with the networking. I quickly figure it out that the network card enumeration (eth0 eth1 swap thingy) is screwed, but it is easy to fix by editing /etc/udev/rules.d/70-persistent-net.rules and removing the lines that refer (by MAC address) to the network card which is in the other machine.

Pelican up and running

2013-03-08T00:00:00+01:00

I decided to start blogging again. I wanted to move away from Blogspot and write posts using some markup in vi and publish them as static HTML. I looked into many static blog generators and I ended up choosing Pelican.

The steps in the documentation worked fine, I had my blog running in about 2 mins. I decided to separate the site generation in two different repositories

https://github.com/wagdav/thewagner.net contains the posts in RST files and the neccessary config files for Pelican.
https://github.com/wagdav/wagdav.github.com contains the actual HTML source and receieves only automatic updates from the first repository. This is set up to generate a User Github Page.

On my laptop I have the clone of these two repositories as:

~/blog/thewagner.net/
~/blog/wagdav.github.com/

I configured Pelican to place its output in ../wagdav.github.com. I needed to make a small adjustment in the generated Makefile, to prevent the make clean command destroying the git repository in the output directory and deleting the files needed for Github Pages. So now the clean target is:

clean:
    @find $(OUTPUTDIR) -mindepth 1 -not -iwholename '*/.git*' \
                                   -not -name 'README.md' \
                                   -not -name 'CNAME' \

which does the job just fine.

The next steps are to import posts from my old blog and change the default style. I want to have something that uses Twitter Bootstrap.

Hello world

2013-03-06T00:00:00+01:00

This is the first post.