2021 Complexity Map

Mike's Notes

A useful diagram from the book Atlas of Social Complexity by Brian Castellani and Lasse Gerrits.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

18/04/2025

2021 Complexity Map

By: Brian Castellani and Lasse Gerrits
Art & Science Factory: 


Storing times for human events

Mike's Notes

Note

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

17/05/2025

Storing times for human events

By: Simon Willison
Simon Willison's Webblog: 27/11/2024

I’ve worked on various event websites in the past, and one of the unintuitively difficult problems that inevitably comes up is the best way to store the time that an event is happening. Based on that past experience, here’s my current recommendation.

This is the expanded version of a comment I posted on lobste.rs a few days ago, which ended up attracting a bunch of attention on Twitter.

The problem

  • The “best practice” that isn’t
  • Things that can go wrong
  • User error
  • International timezone shenanigans
  • Microsoft Exchange and the DST update of 2007
  • My recommendation: store the user’s intent time and the location/timezone
  • Timezone UIs suck, generally

The problem #

An event happens on a date, at a time. The precise details of that time are very important: if you tell people to show up to your event at 7pm and it turns out they should have arrived at 6pm they’ll miss an hour of the event!

Some of the worst bugs an events website can have are the ones that result in human beings traveling to a place at a time and finding that the event they came for is not happening at the time they expected.

So how do you store the time of an event?

The “best practice” that isn’t #

Any time you talk to database engineers about dates and times you’re likely to get the same advice: store everything in UTC. Dates and times are complicated enough that the only unambiguous way to store them is in UTC—no daylight savings or timezones to worry about, it records the exact moment since the dawn of the universe at which the event will take place.

Then, when you display those times to users, you can convert them to that user’s current timezone—neatly available these days using the Intl.DateTimeFormat().resolvedOptions().timeZone browser API.

There’s a variant of this advice which you’re more likely to hear from the PostgreSQL faithful: use TIMESTAMP WITH TIME ZONE or its convenient alias timestamptz. This stores the exact value in UTC and sounds like it might store the timezone too... but it doesn’t! All that’s stored is that UTC value, converted from whatever timezone was active or specified when the value was inserted.

In either case, we are losing critical information about when that event is going to happen.

Things that can go wrong #

What’s wrong with calculating the exact UTC time the event is starting and storing only that?

The problem is that we are losing crucial details about the event creator’s original intent.

If I arrange an evening meetup for next year on December 3rd at 6pm, I mean 6pm local time, by whatever definition of local time is active on that particular date.

There are a number of ways this time can end up misinterpreted:

  • User error: the user created the event with an incorrect timezone
  • User error: the user created the event in the wrong location, and later needs to fix it
  • International timezone shenanigans: the location in which the event is happening changes its timezone rules at some point between the event being created and the event taking place

User error #

By far the most common issue here is user error with respect to how the event was initially created.

Maybe you asked the user to select the timezone as part of the event creation process. This is not a particularly great question: most users don’t particularly care about timezones, or may not understand and respect them to the same extent as professional software developers.

If they pick the wrong timezone we risk showing the wrong time to anyone else who views their event later on.

My bigger concern is around location. Imagine a user creates their event in Springfield, Massachusetts... and then a few days later comes back and corrects the location to Springfield, Illinois.

That means the event is happening in a different timezone. If the user fails to update the time of the event to match the new location, we’re going to end up with an incorrect time stored in our database.

International timezone shenanigans #

One of my favourite niche corners of the internet is the tz@iana.org mailing list. This is where the maintainers of the incredible open source tz database hang out and keep track of global changes to timezone rules.

It’s easy to underestimate how much work this is, and how weird these rule changes can be. Here’s a recent email proposing a brand new timezone: Antarctica/Concordia:

Goodmorning. I’m writing here to propose a new time zone for an all-year open Antarctic base. The base is a French–Italian research facility that was built 3,233 m (10,607 ft) above sea level at a location called Dome C on the Antarctic Plateau, Antarctica. https://en.wikipedia.org/wiki/Concordia_Station

The timezone is UTC+8 without DST.

That’s a pretty easy one. Here’s a much more complicated example from March 2023: Lebanon DST change internally disputed:

Lebanon is going through many internal disputes surrounding the latest decision to delay DST. Many institutions are refusing to comply with the change and are going to adopt regular DST on Sunday Mar 26th. Those institutions include but are not limited to:

  • News agencies
  • Religious organizations
  • Schools, universities, etc...

The refusal is mainly centered the legality of that decision and, obviously, the technical chaos it will create because of its short notice. Moreover, as some of the below articles mention, this is also causing sectarian strife.

Lebanon ended up with more than one timezone active at the same time, depending on which institution you were talking to!

It’s surprisingly common for countries to make decisions about DST with very little notice. Turkey and Russia and Chile and Morocco are four more examples of countries that can often cause short-term chaos for software developers in this way.

If you’ve stored your event start times using UTC this is a big problem: the new DST rules mean that an already-existing event that starts at 6pm may now start at 5pm or 7pm local time, according to the UTC time you’ve stored in your database.

Microsoft Exchange and the DST update of 2007 #

Via fanf on Lobsters I heard about a fascinating example of this problem in action. In 2005 the Bush administration passed the Energy Policy Act of 2005, one part of which updated the rules for when DST would start across most of the USA.

This resulted in a bug where Microsoft Exchange and Outlook would display appointment times incorrectly! From Exchange Server and Daylight Saving Time (DST) 2007:

After installing the DST updates, all old recurring and single instance appointments that occur during the delta period between the DST 2007 rules and the previous DST rules will be one hour later. These appointments will need to be updated so that they will display correctly in Outlook and Outlook Web Access, and for CDO based applications.

Microsoft released a special “Exchange Calendar Update Tool” executable for people to run to fix all of those upcoming calendar events.

My recommendation: store the user’s intent time and the location/timezone #

My strong recommendation here is that the most important thing to record is the original user’s intent. If they said the event is happening at 6pm, store that! Make sure that when they go to edit their event later they see the same editable time that they entered when they first created it.

In addition to that, try to get the most accurate possible indication of the timezone in which that event is occurring.

For most events I would argue that the best version of this is the exact location of the venue itself.

Users may find timezones confusing, but they hopefully understand the importance of helping their attendees know where exactly the event is taking place.

If you have the venue location you can almost certainly derive the timezone from it. I say almost because, as with anything involving time, there are going to be edge-cases—most critically for venues that are exactly on the line that divides one timezone from another.

I haven’t sat down to design my ideal UI for this, but I can imagine something which makes it abundantly clear to the user exactly where and when the event is taking place at that crucial local scale.

Now that we’ve precisely captured the user’s intent and the event location (and through it the exact timezone) we can denormalize: figure out the UTC time of that event and store that as well.

This UTC version can be used for all sorts of purposes: sorting events by time, figuring out what’s happening now/next, displaying the event to other users with its time converted to their local timezone.

But when the user goes to edit their event, we can show them exactly what they told us originally. When the user edits the location of their event we can maintain that original time, potentially confirming with the user if they want to modify that time based on the new location.

And if some legislature somewhere on earth makes a surprising change to their DST rules, we can identify all of the events that are affected by that change and update that denormalized UTC time accordingly.

Timezone UIs suck, generally #

As an aside, here’s my least favorite time-related UI on the modern internet, from Google Calendar:

Google Calendar dialog for Event time zone, has a checkbox for Use separate start and end time zones and then a dropdown box with visible options (GMT-11:00) Niue Time, (GMT-11:00) Samoa Standard Time, (GMT-10:00) Cook Islands Standard Time, (GMT-10:00) Hawaii-Aleutian Standard Time, (GMT-10:00) Hawaii-Aleutian Time, (GMT-10:00) Tahiti Time, (GMT-09:30) Marquesas Time, (GMT-09:00) Alaska Time - Anchorage

There isn’t even a search option! Good luck finding America/New_York in there, assuming you knew that’s what you were looking for in the first place.

tz database

Mike's Notes

I found this great article about the world's time zones. This will need to be added to Pipi in the future.

The code example below is for New York.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

18/04/2025

tz database

Wikipedia:

The tz database is a collaborative compilation of information about the world's time zones and rules for observing daylight saving time, primarily intended for use with computer programs and operating systems. Paul Eggert has been its editor and maintainer since 2005, with the organizational backing of ICANN. The tz database is also known as tzdata, the zoneinfo database or the IANA time zone database (after the Internet Assigned Numbers Authority), and occasionally as the Olson database, referring to the founding contributor, Arthur David Olson.

Its uniform naming convention for entries in the database, such as America/New_York and Europe/Paris, was designed by Paul Eggert. The database attempts to record historical time zones and all civil changes since 1970, the Unix time epoch. It also records leap seconds.

The database, as well as some reference source code, is in the public domain. New editions of the database and code are published as changes warrant, usually several times per year.

...

Move to ICANN

ICANN took responsibility for the maintenance of the database on 14 October 2011. The full database and a description of plans for its maintenance are available online from IANA.

Code example

# Rule  NAME    FROM    TO      TYPE    IN      ON      AT      SAVE    LETTER/S

Rule    US      1918    1919    -       Mar     lastSun 2:00    1:00    D
Rule    US      1918    1919    -       Oct     lastSun 2:00    0       S
Rule    US      1942    only    -       Feb     9       2:00    1:00    W # War
Rule    US      1945    only    -       Aug     14      23:00u  1:00    P # Peace
Rule    US      1945    only    -       Sep     30      2:00    0       S
Rule    US      1967    2006    -       Oct     lastSun 2:00    0       S
Rule    US      1967    1973    -       Apr     lastSun 2:00    1:00    D
Rule    US      1974    only    -       Jan     6       2:00    1:00    D
Rule    US      1975    only    -       Feb     23      2:00    1:00    D
Rule    US      1976    1986    -       Apr     lastSun 2:00    1:00    D
Rule    US      1987    2006    -       Apr     Sun>=1  2:00    1:00    D
Rule    US      2007    max     -       Mar     Sun>=8  2:00    1:00    D
Rule    US      2007    max     -       Nov     Sun>=1  2:00    0       S
....
# Rule  NAME    FROM    TO      TYPE    IN      ON      AT      SAVE    LETTER
Rule    NYC     1920    only    -       Mar     lastSun 2:00    1:00    D
Rule    NYC     1920    only    -       Oct     lastSun 2:00    0       S
Rule    NYC     1921    1966    -       Apr     lastSun 2:00    1:00    D
Rule    NYC     1921    1954    -       Sep     lastSun 2:00    0       S
Rule    NYC     1955    1966    -       Oct     lastSun 2:00    0       S
# Zone  NAME            GMTOFF  RULES   FORMAT  [UNTIL]
Zone America/New_York   -4:56:02 -      LMT     1883 November 18, 12:03:58
                        -5:00   US      E%sT    1920
                        -5:00   NYC     E%sT    1942
                        -5:00   US      E%sT    1946
                        -5:00   NYC     E%sT    1967
                        -5:00   US      E%sT

Interview with James Miller

Mike's Notes

Here is a fascinating video interview I did with my friend James "Jim" Miller on 13 December 2024. Jim lives in Seattle, USA.

He wrote a series of essays, including Universal Evolution, Probability, and Life's Origins, which are available on his SubStack.

I published his essay previously.

Any errors in the interview are due to my still learning how to drive Zoom.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

18/04/2025

Interview with James Miller

By: Mike Peters
Zoom: 13/12/2024

Mike is the inventor and architect of Pipi and the founder of Ajabbi.

Interview

The Developer’s Guide to Fine-Grained Authorization

Mike's Notes

A great article from the WorkOS Blog about Fine-Grained-Authorisation (FGA).

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

17/05/2025

The Developer’s Guide to Fine-Grained Authorization

By: 
WorkOS: 24/10/2024

As apps have become more complex, especially with the rise of user-generated content, the need for a more granular and scalable authorization scheme has become crucial. Unlike other models, Fine-Grained Authorization defines permissions at the resource level, providing precision and the ability to handle millions of authorization requests per second.

Most developers are familiar with RBAC, or Role-Based Access Control – it’s the old school standard for how authorization is built. Each user has a role, each app section has a permission, and you check on every action or page load to make sure that the currently authenticated user has the right to do what they’re doing. 

But as apps have gotten more complicated – and supported more user generated content – teams have been focusing on a newer, more granular, and more scalable authorization scheme: Fine Grained Authorization. Fine Grained Authorization (FGA) defines permissions on a resource basis – individual users having access to individual resources. You can define precisely who can see or do what, down to individual fields in a database or specific actions in an application. 

FGA is powerful – it supports millions of authorization requests per second for apps like YouTube and Google Drive – but it’s also notoriously difficult to implement yourself. This post will walk through what you need to know to get it done, from your data model to 3rd party options and UI considerations.

‍FGA basics – the data model

At the core of FGA is the relationship between a user and a resource. A resource is what the canonical object in your application is: a cap table for Carta, a bank account for J.P. Morgan, or a document in Google Drive, or a file in Figma.



The relationship field is highly dependent on your business: it could be as simple as read or write or as complex as edit_maximum_of_5_rows_during_the_day. This type of FGA relationship data is very difficult to represent efficiently in a relational database, and tends to be a much better fit for a graph model.

In practice, you don’t need to enumerate every possible permutation of user and resource in your system. FGA systems will also make use of roles, especially to assume default states. For example, you might have a group called “Airtable Admins” who have default permissions to every single Airtable base in the organization. In essence, people like to define their permissions through groups and roles, but check them at the user level.

Building resolver logic

Where the FGA data model gets complicated is what we call “resolvers” – all of the cases where a relationship isn’t explicitly coded into an authorization check, but is implied by other existing roles or hierarchies. You will need to write custom logic to resolve these yourself. Let’s run through a few examples. 

Consider some basic hierarchy:

  • You are part of the Airtable Admins group
  • The Airtable Admins group is an owner of Top Secret Base
  • Ergo, you are an owner of Top Secret Base

In an RBAC system, you being in the Airtable Admins group would mean that you pass the check to own Top Secret Base. In an FGA world though, it goes a level deeper, almost like a compiler or resolver: because you are in the Airtable Admins group, that means that you are personally an owner of Top Secret Base, and that’s why you pass the check. So this is logic that you need to write.

Another example: imagine a simple Google Docs authorization hierarchy. I set access to my doc such that everyone in my organization has comment access. Even though I haven’t shared the doc individually with my coworker, since she is part of my organization, my system needs to calculate that she, individually, has comment access to the doc. This kind of resolver is simple enough to build.

Next part. A Google Doc has the concept of an owner, an editor, and a viewer. An owner implies that you’re also an editor, which implies that you’re a viewer. This notion of permission inheritance is another “resolver” related thing that you will need to build yourself.

These two ideas – inheritance between roles, and inheritance between permissions – are the two big pieces of building a resolver system. If you can get these implemented, most other cases will fall in line. But you will likely find that edge cases from your customers make these things take much longer than they should, especially at scale.

Centralized vs. decentralized FGA

A central piece of discussion in authorization is building centralized systems vs. decentralized ones – should your authorization system be its own service? There are two pieces to this: the data storage can be centralized or decentralized, and your check system can be too. 

In an RBAC world, most teams don’t need to worry about services for a while. The scale of data is inherently smaller, you can scale to even millions of rows without much of a hiccup, and many developers are just shoving roles and permissions into a JWT anyway. FGA, on the other hand, tends to lend itself more to a centralized architecture (i.e. its own service) for a couple of reasons. 

The first is that if your customers have the kinds of complex requirements that call for building out an FGA system, chances are you’re further along in your company journey and might have already split out your architecture into a service based one. And you certainly don’t want to replicate your check logic for every single service.

The second is that the FGA data model is inherently more inclined towards large data volumes than RBAC, and if you’re building it yourself, at some point you will necessarily need to move it into its own database so it can scale independently. This, plus the fact that these relationships are more naturally represented in a graph database, which in all likelihood is not the kind of database you’re using for production.

In practice, whether teams choose to centralize or decentralize this tends to depend more on their existing architecture. But FGA definitely lends itself more to a centralized implementation.

Outsourcing FGA (open source or otherwise)

Tons of teams will implement RBAC on their own. FGA, given how difficult it is, less common. And until a few years ago, there weren’t a ton of options for outsourcing.

Google Zanzibar is Google’s internal authorization system: it supports millions of authorization requests per second for products like YouTube and Drive. In 2019, they released a paper at USENIX that ran through how it works. To quote Carta’s AuthZ blog post, which has a nice summary:

One of Zanzibar’s core features is a uniform language that is used to define permissions. Zanzibar consumers use the uniform language to build Access Control Lists (ACLs). ACLs are like unix file permissions. They give users access to individual resources in the system. With Zanzibar, services compose abstractions for user permission groups. User permission groups can compose each other. They also grant access to low-level resource ACLs.

So essentially, it’s implemented as a centralized service that handles both the data and the resolvers. Since the 2019 paper, several open and closed source implementations have popped up on the market. You now have a bunch of options for “buying” FGA, like:

  • WorkOS FGA (built on Warrant)
  • SpiceDB (OSS) from Authzed
  • OpenFGA (OSS)

You can pick OSS FGA systems off the shelf, but it only solves a part of the problem, since at scale these things are major operations to run. If you’re at a company like Figma, you’ll still need a 3 person team to manage this for you. You need to provide and manage a database, maintain performance, and keep the high uptime and availability that a centralized system needs.

I would also be remiss if I didn’t mention Open Policy Agent. It’s a framework for writing and evaluating policies for authorization. But OPA doesn’t have any data; it’s just policies. It’s like you’re running a function, but you need to provide the input at runtime. So I wouldn’t consider it a solution (in the traditional sense of the word) for completely building FGA. The Zanzibar model is a more natural fit for these business app type use cases, whereas OPA is really good for infra authorization (e.g. only allow ingress from this IP). 

Frontend / UI considerations

FGA has a few considerations to mention when it comes to building UI for your users to set and adjust these permissions. You’ve probably seen this bad boy before:



In an FGA universe, your users need to be able to share any resource with any individual user or group, plus see (and adjust) which users currently have access. That’s this screen:



On the backend, this expresses itself as two different types of queries. There’s the concept of the check, which deals with whether (in a boolean sense) a user has permission to do a certain thing to a certain resource. But there is also the concept of the list, which deals with the total set of who has what permissions to this resource. The list is the data you want when you’re building UI for users to be creating and updating these permissions in the first place.

In WorkOS we have these two APIs split out. There’s the Check API, which you’d use to do a check on whether a user has access to something, and the Query API, which you’d use to see all of the users that have access to something. 

Bonus: FGA and Identity Providers

If you have customers large and sophisticated enough to be requesting authorization features that have you wondering if you should look at FGA, there’s a 90% chance they will be running their identity and access management through an Identity Provider (IdP) like Okta. These IdPs are the central source of truth for who works at these organizations, how they authenticate, and what they have access to.

RBAC plays nicely with these IdPs; FGA does not. The fundamental concept of resource-based authorization doesn’t really work well with IdP-based authorization at all. FGA is dynamic: I just created a new Figma file, I just built a new Hubspot contact list, and here’s a project ID. There’s no way an IT admin would be able to interpret all of these and apply the appropriate groups and roles. 

We are guessing that things will go towards a hybrid RBAC/FGA direction in the future. If you’re interested in learning more about how these ideas interact with IdPs, check out our Developer’s Guide to RBAC and IdPs. It runs through a few best practices for setting up the FGA/role architecture on your end to work well with Okta and the like (to the extent it’s currently possible).

Eruption Forecasting using Bayesian Networks

Mike's Notes

I don't know enough to describe Pipi technically. I know that I built Fuzzy Logic and Markov into its design. I suspect Bayesian Networks and some kind of Monte Carlo are also involved. But I don't have any formal training in Mathematics apart from High School, to be sure.

Part of the problem is that I accidentally stumbled across an architecture that worked to solve a problem I wanted to solve.

I will ask my friends Alex and Chris ( both retired computer scientists) to help determine what is happening.

I may also find a friendly mathematician lurking around Wolfram.

Here is an article from the GNS publication "Beneath the Waves" that makes me think Bayesian probably is happening because of the first figure. It looks like what I have done anyway.

"Bayesian networks are a type of Probabilistic Graphical Model that can be used to build models from data and/or expert opinion. They can be used for a wide range of tasks including diagnostics, reasoning, causal modeling, decision making under uncertainty, anomaly detection, automated insight and prediction." ... Bayes Server

"A Bayesian network (also known as a Bayes network, Bayes net, belief network, or decision network) is a probabilistic graphical model that represents a set of variables and their conditional dependencies via a directed acyclic graph (DAG).[1] While it is one of several forms of causal notation, causal networks are special cases of Bayesian networks. Bayesian networks are ideal for taking an event that occurred and predicting the likelihood that any one of several possible known causes was the contributing factor. For example, a Bayesian network could represent the probabilistic relationships between diseases and symptoms. Given symptoms, the network can be used to compute the probabilities of the presence of various diseases." ... Wikipedia

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

17/05/2025

Eruption Forecasting using Bayesian Networks

GNS Science: 04/12/2024

A novel method of eruption forecasting has been developed using Bayesian Networks. This innovation allows for data-based forecasts of activity without reliance on a single monitoring dataset.

What is a Bayesian Network?

A Bayesian Network is a simple model and statistical tool that can be used to determine the probability of an event happening, based on:

  • a set of variables (nodes) that influence the event occurring,
  • the relationships and dependencies between the variables,
  • prior information about the event, and
  • prior information about the variables.

Bayesian networks can combine different sources of information by defining a system of conditional probabilities between sources. An event is then simply a certain constellation of that system for which the Bayesian Network will give us a probability of occurrence.

A Bayesian network (e.g. see Figure 1) has two main components:

  • the causal relationships between the variables in the system, and
  • the actual probabilities for the variable (node) that are used to make predictions.



Figure 1: A Bayesian Network showing the relationship between nodes for a volcanic eruption and the variables that can be measured (e.g. gas monitoring (CO2, H2S, SO2), volcanic tremors (RSAM)).

A Bayesian network is learned from data and previous patterns. Having knowledge of a previous event can help the model to predict the probability of a future event. Because Bayesian Networks also include prior information, they  are resilient to gaps in information (such as missing sensors).

Eruption Forecasting at Whakaari | White Island

Since the 2019 eruption, destruction of monitoring equipment on island means that Whakaari (White Island) has reduced real-time monitoring data. Yet, eruption forecasting is still possible through the ongoing monitoring of volcanic activity (e.g. remote sensing, monthly gas plume monitoring). Using Bayesian Networks, we have developed and tested a novel method of eruption forecasting that does not solely rely on on-island data streams.

The Bayesian Network was trained on most of the Whakaari monitoring data available since 2009. Figure 2 shows the timing of past eruptions on Whakaari. Group A data (2009-2012) was the initial training set used to model the 2013 eruption, Group A & B were used to predict the 2016 eruption, and so forth. In this manner, the error margins of the network were greatly improved, and the model was found to be effective even when not all data streams are available, such as the current day situation. Results from this network showed increases in eruption likelihood prior to the 2012-2019 sequences, even when using airborne gas measurements only.


Figure 2: Past monitoring data was used to train the Whakaari Bayesian Network.

Once this new method has gone through the scientific peer-review process, it will become an important part in setting life safety parameters around the volcano. Forecasts will be updated automatically as new monitoring data becomes available and provided to geohazard response staff. This information will help GNS Science to perform it’s monitoring and advisory requirements to advise external stakeholders of increases in eruption likelihood.

Forecasts from our BN are easy to understand and interpret by volcanologists, which adds to its usefulness for decision support and expert advice. Expert judgment can also be easily integrated and potentially help to improve forecasts in the absence of data.

Comparison of open-source configuration management software

Mike's Notes

I need to think about how to manage complex cloud deployments, both public and private.

Parts of Pipi will need to be deployed in virtual machines (VM) on GCP, Azure, AWS, IBM, etc.

OpenNebula would be a good start, initially with a free community edition and then a paid subscription as needs develop.

Using text configuration files would enable Pipi to self-manage the process. Ansible Playbooks written in YAML could be a solution.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

17/05/2025

Comparison of open-source configuration management software

Wikipedia:

...

Short descriptions

Not all tools have the same goal and feature set. Here is a short description of each software package to help distinguish between them.

Ansible

Combines multi-node deployment, ad-hoc task execution, and configuration management in one package. Manages nodes over SSH and requires python (2.6+ or 3.5+) to be installed on them.[105] Modules work over JSON and standard output and can be written in any language. Uses YAML to express reusable descriptions of systems.

Bcfg2

Software to manage the configuration of a large number of computers using a central configuration model and the client–server paradigm. The system enables reconciliation between clients' state and the central configuration specification. Detailed reports provide a way to identify unmanaged configuration on hosts. Generators enable code or template-based generation of configuration files from a central data repository.

CFEngine

Lightweight agent system. Manages configuration of a large number of computers using the client–server paradigm or stand-alone. Any client state which is different from the policy description is reverted to the desired state. Configuration state is specified via a declarative language.[106] CFEngine's paradigm is convergent "computer immunology".[107]

cdist

cdist is a zero dependency configuration management system: It requires only ssh on the target host, which is usually enabled on all Unix-like machines. Only the administration host needs to have Python 3.2 installed.

Chef

Chef is a configuration management tool written in Erlang,[108] and uses a pure Ruby DSL for writing configuration "recipes". These recipes contain resources that should be put into the declared state. Chef can be used as a client–server tool, or used in "solo" mode.[109]

Consfigurator

While Debian and derivatives are the best supported distributions, Consfigurator also work on other distributions and various unixes but they have less support for properties for configuring specific aspects of the system. Consfigurator can set properties to be applied in scheme. This requires Consfigurator to be installed on the target computer. A more restricted language is also available which works without needing Consfigurator to be installed on the target. Remote configuration is also supported: the of hosts can be defined with scheme code.

Guix

Guix integrates many things in the same tool (a distribution, package manager, configuration management tool, container environment, etc). To remotely manage systems, it needs the target machines to already run Guix or it can also alternatively deploy configurations inside Digital Ocean Droplet. The machines are configured with Scheme.

ISconf

Tool to execute commands and replicate files on all nodes. The nodes do not need to be up; the commands will be executed when they boot. The system has no central server so commands can be launched from any node and they will replicate to all nodes.

Juju

Juju concentrates on the notion of service, abstracting the notion of machine or server, and defines relations between those services that are automatically updated when two linked services observe a notable modification.

Local Configuration system (LCFG)

LCFG manages the configuration with a central description language in XML, specifying resources, aspects and profiles. Configuration is deployed using the client–server paradigm. Appropriate scripts on clients (called components) transcribe the resources into configuration files and restart services as needed.

Open PC server integration (Opsi)

Opsi is desktop management software for Windows clients based on Linux servers. It provides automatic software deployment (distribution), unattended installation of OS, patch management, hard- and software inventory, license management and software asset management, and administrative tasks for the configuration management.

PIKT

PIKT is foremost a monitoring system that also does configuration management. "PIKT consists of a sophisticated, feature-rich file preprocessor; an innovative scripting language with unique labor-saving features; a flexible, centrally directed process scheduler; a customizing file installer; a collection of powerful command-line extensions; and other useful tools."

Puppet

Puppet consists of a custom declarative language to describe system configuration, distributed using the client–server paradigm (using XML-RPC protocol in older versions, with a recent switch to REST), and a library to realize the configuration. The resource abstraction layer enables administrators to describe the configuration in high-level terms, such as users, services and packages. Puppet will then ensure the server's state matches the description. There was brief support in Puppet for using a pure Ruby DSL as an alternative configuration language starting at version 2.6.0. However this feature was deprecated beginning with version 3.1.[

Quattor

The quattor information model is based on the distinction between the desired state and the actual state. The desired state is registered in a fabric-wide configuration database, using a specially designed configuration language called Pan for expressing and validating configurations, composed out of reusable hierarchical building blocks called templates. Configurations are propagated to and cached on the managed nodes.

Radmind

Radmind manages hosts configuration at the file system level. In a similar way to Tripwire (and other configuration management tools), it can detect external changes to managed configuration, and can optionally reverse the changes. Radmind does not have higher-level configuration element (services, packages) abstraction. A graphical interface is available (only) for OS X.

Rex

Rex is a remote execution system with integrated configuration management and software deployment capabilities. The admin provides configuration instructions via so-called Rexfiles. They are written in a small DSL but can also contain arbitrary Perl. It integrates well with an automated build system used in CI environments.

Salt

Salt started out as a tool for remote server management. As its usage has grown, it has gained a number of extended features, including a more comprehensive mechanism for host configuration. This is a relatively new feature facilitated through the Salt States component. With the traction that Salt has gotten in the last bit, the support for more features and platforms might continue to grow.

SmartFrog

Java-based tool to deploy and configure applications distributed across multiple machines. There is no central server; you can deploy a .SF configuration file to any node and have it distributed to peer nodes according to the distribution information contained inside the deployment descriptor itself.

Spacewalk

Spacewalk is an open source Linux and Solaris systems management solution[buzzword] and is the upstream project for the source of Red Hat Network Satellite. Spacewalk works with RHEL, Fedora, and other RHEL derivative distributions like CentOS, Scientific Linux, etc. There are ongoing efforts on getting it packaged for inclusion in Fedora. Spacewalk provides systems inventory (hardware and software information, installation and updates of software, collection and distribution of custom software packages into manageable groups, provision systems, management and deployment of configuration files, system monitoring, virtual guest provisioning, starting/stopping/configuring virtual guests and delegating all of these actions to local or LDAP users and system entitlements). As of May 2020, Spacewalk is now EOL with users having moved to either Uyuni or Foreman/Katello.

STAF

The Software Testing Automation Framework (STAF) enables users to create cross-platform, distributed software test environments. STAF removes the tedium of building an automation infrastructure, thus enabling users to focus on building their automation solution.[buzzword] The STAF framework provides the foundation upon which to build higher-level solutions[buzzword], and provides a pluggable approach supported across a large variety of platforms and languages.

Synctool

Synctool aims to be easy to understand, learn and use. It is written in Python and makes use of SSH (passwordless, with host-based or key-based authentication) and rsync. No specific language is needed to configure Synctool. Synctool has dry run capabilities that enable surgical precision. Synctool depends on Python2 which is now EOL and there are no current plans to migrate it to Python3.

OpenNebula Conference

Mike's Notes

This might be very useful later for Pipi to deploy onto server farms. OpenNebula is an excellent alternative to VMware.

There is a free community edition and paid subscriptions. An enterprise subscription looks promising. Managed services are also available.

Playbooks

It uses playbooks which can drive Ansible when deploying. 

"Playbooks are YAML files that store lists of tasks for repeated executions on managed nodes. Each Playbook maps (associates) a group of hosts to a set of roles. Each role is represented by calls to Ansible tasks." - Wikipedia

Example of a playbook code at the bottom

Options

It can do "heterogeneous data centre, public cloud and edge computing infrastructure resources".

Conferences

"Born back in 2013, OpenNebula Conferences are educational events that serve as a meeting point of cloud users, developers, administrators, integrators and researchers, featuring talks with experiences and use cases. They also include Hands-on tutorials, workshops, and hacking sessions that provide an opportunity to discuss burning ideas, and meet face to face to discuss development. Previous speakers include Telefonica, Booking.com, Innologica, King, Nordeus, StorPool, Santander Bank, CentOS, European Space Agency, FermiLab, Puppet, Red Hat, BlackBerry, Akamai, Runtastic, Citrix, Trivago… and many more." ... OpenNebula

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

17/05/2025

OpenNebula

Wikipedia:

OpenNebula is an open source cloud computing platform for managing heterogeneous data center, public cloud and edge computing infrastructure resources. OpenNebula manages on-premises and remote virtual infrastructure to build private, public, or hybrid implementations of infrastructure as a service (IaaS) and multi-tenant Kubernetes deployments. The two primary uses of the OpenNebula platform are data center virtualization and cloud deployments based on the KVM hypervisor, LXD/LXC system containers, and AWS Firecracker microVMs. The platform is also capable of offering the cloud infrastructure necessary to operate a cloud on top of existing VMware infrastructure. In early June 2020, OpenNebula announced the release of a new Enterprise Edition for corporate users, along with a Community Edition. OpenNebula CE is free and open-source software, released under the Apache License version 2. OpenNebula CE comes with free access to patch releases containing critical bug fixes but with no access to the regular EE maintenance releases. Upgrades to the latest minor/major version is only available for CE users with non-commercial deployments or with significant open source contributions to the OpenNebula Community. OpenNebula EE is distributed under a closed-source license and requires a commercial Subscription.

...

OpenNebula orchestrates storage, network, virtualization, monitoring, and security technologies to deploy multi-tier services (e.g. compute clusters) as virtual machines on distributed infrastructures, combining both data center resources and remote cloud resources, according to allocation policies. According to the European Commission's 2010 report "... only few cloud dedicated research projects in the widest sense have been initiated – most prominent amongst them probably OpenNebula ...".

The toolkit includes features for integration, management, scalability, security and accounting. It also claims standardization, interoperability and portability, providing cloud users and administrators with a choice of several cloud interfaces (Amazon EC2 Query, OGF Open Cloud Computing Interface and vCloud) and hypervisors (VMware vCenter, KVM, LXD/LXC and AWS Firecracker), and can accommodate multiple hardware and software combinations in a data center.

OpenNebula is sponsored by OpenNebula Systems (formerly C12G).

OpenNebula is widely used by a variety of industries, including cloud providers, telecommunication, information technology services, government, banking, gaming, media, hosting, supercomputing, research laboratories, and international research projects[citation needed].

...

Internal architecture

Basic components

OpenNebula Internal Architecture

  • Host: Physical machine running a supported hypervisor.
  • Cluster: Pool of hosts that share datastores and virtual networks.
  • Template: Virtual Machine definition.
  • Image: Virtual Machine disk image.
  • Virtual Machine: Instantiated Template. A Virtual Machine represents one life-cycle, and several Virtual Machines can be created from a single Template.
  • Virtual Network: A group of IP leases that VMs can use to automatically obtain IP addresses. It allows the creation of Virtual Networks by mapping over the physical ones. They will be available to the VMs through the corresponding bridges on hosts. Virtual network can be defined in three different parts:
    1. Underlying of physical network infrastructure.
    2. The logical address space available (IPv4, IPv6, dual stack).
    3. Context attributes (e.g. net mask, DNS, gateway). OpenNebula also comes with a Virtual Router appliance to provide networking services like DHCP, DNS etc.

Playbook example (YAML)

(one-deploy-py3.12) front-end:~/my-one$ ansible-playbook -v opennebula.deploy.main

Using /home/basedeployer/my-one/ansible.cfg as config file

running playbook inside collection opennebula.deploy

[WARNING]: Could not match supplied host pattern, ignoring: bastion


PLAY [bastion] *******************************************************************************************

skipping: no hosts matched

[WARNING]: Could not match supplied host pattern, ignoring: grafana

[WARNING]: Could not match supplied host pattern, ignoring: mons

[WARNING]: Could not match supplied host pattern, ignoring: mgrs

[WARNING]: Could not match supplied host pattern, ignoring: osds


PLAY [frontend,node,grafana,mons,mgrs,osds] **************************************************************


TASK [opennebula.deploy.helper/python3 : Bootstrap python3 intepreter] ***********************************

skipping: [f1] => changed=false

  attempts: 1

  msg: /usr/bin/python3 exists, matching creates option

skipping: [n2] => changed=false

  attempts: 1

  msg: /usr/bin/python3 exists, matching creates option

skipping: [n1] => changed=false

  attempts: 1

  msg: /usr/bin/python3 exists, matching creates option


...


TASK [opennebula.deploy.prometheus/server : Enable / Start / Restart Alertmanager service (NOW)] *********

skipping: [f1] => changed=false

  false_condition: features.prometheus | bool is true

  skip_reason: Conditional result was False


PLAY [grafana] *******************************************************************************************

skipping: no hosts matched


PLAY RECAP ***********************************************************************************************

f1                         : ok=84   changed=33   unreachable=0    failed=0    skipped=75   rescued=0    ignored=0

n1                         : ok=37   changed=12   unreachable=0    failed=0    skipped=57   rescued=0    ignored=0

n2                         : ok=37   changed=12   unreachable=0    failed=0    skipped=48   rescued=0    ignored=0

Life as No One Knows it

Mike's Notes

Astrobiologist Sara Imari Walker takes up complex and abiding questions in Life as No One Knows It: The Physics of Life’s Emergence.


Below is another wonderful article by Maria Popova republished from The Marginalian.

Resources

References

  • No One Knows It: The Physics of Life’s Emergence

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

17/05/2025

The Great Blind Spot of Science and the Art of Asking the Complex Question the Only Answer to Which Is Life

By: Maria Popova
The Marginalian: 4 November 2024

“Real isn’t how you are made… It’s a thing that happens to you,” says the Skin Horse — a stuffed toy brought to life by a child’s love — in The Velveteen Rabbit. Great children’s books are works of philosophy in disguise; this is a fundamental question: In a reality of matter, what makes life alive? A generation later, the Ukrainian Jewish writer Vasily Grossman answered with a deeply original proposition: that life is best defined as freedom, that freedom is the boundary between inanimate matter and animacy.

To me, freedom is the boundary condition where matter reaches for meaning — life, after all, is the only component of the universe free to comprehend the rest. And yet all of our technologies of thought have so far failed to discern what life actually is, how it emerged from non-life, and what to look for when we are looking for it in our laboratories and in the great unfolding experiment that is the universe itself. We have sequenced the human genome and discovered the “God particle,” yet genetics and particle physics have found no common language for communicating and harmonizing their respective discoveries to address the complex question the single answer to which is life.

A century ago, the philosopher Simone Weil admonished against this fragmentation of the problem of reality into parochial questions addressed by disjointed scientific disciplines — “villages” of thought, she called them — each too blinded by its own axioms to make headway on illuminating the whole. “The villagers seldom leave the village,” she wrote. Watching her mathematician brother — the number theory pioneer André Weil — try to reduce the problem of reality to his own science, watching the founding fathers of quantum mechanics do the same, she lamented: “The state of science at a given moment is nothing else but… the average opinion of the village of scientists [who] affirm what they believe they ought to affirm.”

An epoch later, the villages have drifted so far apart as to grow foreign to each other. Gravitational waves, radioactivity, and DNA belong to the same reality — the reality that made life possible — and yet cosmology, chemistry, and biology are too mute to each other to make sense of the deeper meaning behind their respective discoveries. We are still left wondering how reality happens unto life and how life becomes reality.

Astrobiologist Sara Imari Walker takes up these complex and abiding questions in Life as No One Knows It: The Physics of Life’s Emergence (public library).

Trained as a theoretical physicist and disenchanted with her discipline’s insistence that life is a conceptually banal scientific problem subservient to the fundamentals of space, time, energy, and matter, she holds modern physics accountable for providing “a fundamental description of a universe devoid of life” — that is, a description of the universe that negates the very existence of its describers, we who are very much alive. She writes:

We cannot see ourselves clearly because we have not built a theory of physics yet that treats observers as inside the universe they are describing.

In this quest to understand ourselves and the universe that made us, she argues, the vitalists of the eighteenth century — who believed that a concrete non-physical element, a “vital spark,” grants life its aliveness — were no more misguided than the modern materialists who believe that life — that poetry, that whale song, that love — is just a property of physical matter. Reckoning with a colleague’s startling remark that “life does not exist,” she considers the deeper logic beneath this koan-like formulation of the great scientific blind spot of our time:

What modern science has taught us is that life is not a property of matter… There is no magic transition point where a molecule or collection of molecules is suddenly “living.” Life is the vaporware of chemistry: a property so obvious in our day-to-day experience — that we are living — is nonexistent when you look at our parts. If life is not a property of matter, and material things are what exist, then life does not exist.

(And of course, none of it had to exist at all. Life seems to be the imperative of the unnecessary. Long before modern physics, Darwin marveled at how, on this planet shaped by unfeeling forces and moved by fixed laws, “from so simple a beginning endless forms most beautiful and most wonderful have been and are being evolved.” Here was a biologist trained as a geologist shining a sidewise gleam on a cosmological question — a rare vagabond between the villages of science, from a time before they had become separate continents of thought.)

At the heart of the book is the rigorous, passionate insistence that we need a softer and more elastic explanatory membrane between the three hard problems of reality: the hard problem of consciousness (rooted in the mystery of qualia, that inarticulable essence of what it feels like to be oneself, the felt interiority of being alive in a particular embodiment and enmindment), the hard problem of matter (the fact that everything observable arises from the interaction of particles and forces), and the hard problem of life (sculpted of information and an observer of information). Sara writes:

Cast in this way, all three hard problems become one more fundamental problem we cannot seem to avoid any more than we can seem to answer it: Why do some things exist (or experience existence) and not others? It is perhaps the most perplexing question of our existence that anything should exist at all. And if something exists, then why not everything?

By contracting the pinhole of our scrutiny to the question of life, she intimates, we might be able to begin extrapolating an answer to this largest of questions — something that calls not only for new principles but for a new theory of physics and a dismantling of disciplinary boundaries. A century after Weil, Sara points to the same paradox standing between the life of science and the science of life in our own time:

We don’t yet have a general understanding of the category of things that we should group together and call “life.” Therefore either our categorization is wrong or life is not something to be categorized.

[…]

We cannot always see this clearly because of the arbitrary boundaries we set between the current classification of disciplines we think are needed to solve the problem, which are based on paradigms not suited for solving what life is.

Observing that “the boundary between the phenomena we want to think of as life and not life is fuzzy at best and may not exist at all,” she considers the present state of our disciplinary parochialism:

Biologists approach the problem by defining life in terms of observed features of life on Earth, which is not especially useful when you’re looking for life’s origins or for life elsewhere in the universe. Astrobiologists need guiding principles to inform how they conduct their search, but they, too, end up being overly anthropocentric in their reasoning: their search is most often directed at signs of life that would indicate biology exactly as we observe it here on Earth. Chemists either think life does not exist or that it is all chemistry (probably these are equivalent views). Computer scientists tend to focus too much on the software — the information processing and replicative abilities of life — and not enough on the hardware, i.e., the fact that life is a physical system that emerges from chemistry, and that the properties of chemistry literally matter. Physicists tend to focus too much on the physical — life is about thermodynamics and flows of energy and matter — and miss the informational and evolutionary aspects that seem to be the most distinctive features of the things we want to call life. Philosophers focus too much on the need for a definition or the flaws of providing one, and not enough on how we can move as a community beyond the definitional phase into a new paradigm.

Nature does not share these boundaries between disciplines. They are artifacts of our human conception of nature, our need to classify things, and historical contingencies in how our understanding of the reality around us has evolved over the last few centuries. That is, they are the product of paradigms established in the past. We are in part pre-paradigmatic in understanding life as a general phenomenon in the universe because there is no defined discipline that can fully accommodate the intellectual discussion that needs to be had about what life is.

The solution to the unsolved problem of life, she argues, may not be one of new evidence but one of new explanation, just as we watched the planets move for eons before we discerned the laws of their motion to concede a heliocentric universe. Without a clear explanatory model for life here on Earth, she argues, we might never be able to detect life on other worlds — the central task of her own science. With an eye to how the new science of plant intelligence deepens the mystery of what a mind is, Sara considers what kindred blind spots may be afflicting astrobiology:

Plants are just one example that makes clear how the boundary of our imagination does not even intersect with what it is to be among the other multicellular life that surrounds us on this planet.

If we cannot even shift our reference frame enough to understand what it is like to be other inhabitants of our own planet, how could we possibly imagine the truly alien? “Truly alien” here should be understood as other life that does not share any ancestry with our own: that is, that has an entirely unique history with an independent origin. There are no aliens on Earth because as far as we know, all the life we have encountered shares a common history. Even artificial intelligences — sometimes described as alien, are not alien; they are trained on human data, which is itself the product of nearly four billion years of evolution on Earth. AI is as much a part of life on Earth as any of the biological organisms that have evolved here.

A century and a half after the Victorian visionary Samuel Butler presaged the emergence of a new “mechanical kingdom” extending the kingdoms of biological life into our machines, Sara argues that our mechanical and algorithmic creations may not only alter the definition of life but help illuminate its origins:

The emergence of a technosphere may be precisely what is required for a biosphere to solve its own origins and therefore to discover others like it. To make this transition and make first contact, it may be critical to where we sit now in time that we recognize how thinking technologies are the next major transition in the planetary evolution of life on Earth. It is what we might expect as societies scale up and become more complex, just as life simpler than us has done in the past. The functional capabilities of a society have their deepest roots in ancient life, a lineage of information that propagates through physical materials. Just as a cell might evolve along a specific lineage into a multicellular structure (something that’s not inevitable but has happened independently on Earth at least twenty-five times), the emergence of artificial intelligences and planetary-scale data and computation can be seen as an evolutionary progression — a biosphere becoming a technosphere.

“Wherever life can grow, it will. It will sprout out, and do the best it can,” Gwendolyn Brooks wrote in one of her finest, least known poems. A proper understanding of life, Sara argues, must account for that fact — for the tenacity with which life not only continues to exist despite the infinitely greater odds of nonexistence (which anchored Richard Dawkins’s wonderful counterintuitive insistence on the luckiness of death) but continues to exist in its particularity despite the infinitely many other possible configurations. She writes:

If we are ever to understand what life really is, we need to recognize that among the unimaginably large number of things that could exist, or even the smaller subset of ones that we can imagine, only an infinitesimal fraction ever will. Things come into existence when and where it is possible to — and what we call life is the mechanism for making specific things possible when the possibility space is too large for the universe to ever explore all of it.

Out of this arises a crucial distinction between life and being alive (highlighted in the biological fact that most of you is dead). Nearly a century after cybernetics pioneer Norbert Wiener made the then-radical assertion that “we are not stuff that abides, but patterns that perpetuate themselves,” Sara adds:

DNA cannot exist unless there is a physical system (e.g., a cell) with memory of the steps to assemble it. All objects that require information to specify their existence constitute “life.” Life is the high-dimensional combinatorial space of what is possible for our universe to build that can be selected to exist as finite, distinguishable physical objects. Being “alive,” by contrast, is the trajectories traced through that possibility space. The objects that life is made of and that it constructs exist along causal chains extended in time; these lineages of information propagating through matter are what it is to be “alive.” Lineages can assemble individual objects, like a computer, a cup, a cellular membrane, or you in this very instant, but it is the temporally extended structure that is alive. Even over your lifetime you are alive because you are constantly reconstructing yourself — what persists is the informational pattern over time, not the matter.

[…]

The fundamental unit of life is not the cell, nor the individual, but the lineage of information propagating across space and time. The branching pattern at the tips of this structure is what is alive now, and it is what is constructing the future on this planet.

In the remainder of Life as No One Knows It, Sara goes on to explore assembly theory — a new framework for understanding the complexity of living organisms by discerning the minimal number of steps required to assemble them from the most fundamental building blocks — as a possible solution to the abiding problem of what we are. Complement it with pioneering biologist Ernest Everett Just — one of the first scientists to consider this question holistically — on what makes life alive, then revisit Meghan O’Gieblyn on our search for meaning in the age of AI and Alan Turing’s favorite boyhood book about the strange science of how alive you really are.