On a Sandy Beach

Real-world Engineering Challenges #8: Breaking up a Monolith

Mike's Notes

This detailed article by Gergely Orosz is partially copied from his newsletter, The Pragmatic Engineer.

Resources

References

Reference

Repository

Home > Ajabbi Research > Library > Subscriptions > The Pragmatic Engineer
Home > Handbook > DevOps

Last Updated

05/05/2025

Real-world Engineering Challenges #8: Breaking up a Monolith

By: Gergely Orosz

The Pragmatic Engineer: 08/03/2023

Writing The Pragmatic Engineer. Previously at Uber, Skype, Microsoft. Author of The Software Engineer's Guidebook.

‘Real world engineering challenges’ is a series in which I interpret interesting software engineering and engineering management case studies by tech companies. You might learn something new in these articles, as we dive into the concepts they contain.

In most real-world engineering challenges articles, we look at several interesting case studies from tech companies, but today’s issue is different. We’re covering a single case study, but in more detail than previously in this series.

Specifically, we’re diving into a massive migration project by Khan Academy, involving moving one million lines of Python code and splitting them across more than 40 services, mostly in Go, as part of a migration that took 3.5 years and involved around 100 software engineers.

Khan Academy has an interesting engineering blog where I came across several posts with details about how this migration progressed. I got interested in learning more about this journey and so reached out to engineering manager Brian Genisio and former principal software architect Kevin Dangoor. Both Brian and Kevin played key roles in this migration and they have generously shared their inside takes on how the migration played out.

In today’s issue, we reveal new details about this migration, covering:

Context and motivation. Why decide to migrate a very large monolith that is powering a site with heavy traffic? The trigger was the end of Python 2, but there were other reasons, too.
Kickoff and phase. Additionally, the concept of MVE (minimum viable experience.)
The migration phase. The migration approach to each of the fields and APIs.
Things that worked well. Incremental shipping, the side-by-side migration approach, and treating the project as fixed scope, fixed timeline.
Choices worth reconsidering. Was Go the best choice? It’s one of the questions the team still thinks about.
Learnings from working on a 3.5 year-long migration. The extra motivation which hard deadlines provide.

As with all case studies, I write about a given company because I find the topic interesting. I have no financial motivation for this: I don’t get paid, directly or indirectly. In the very rare case when there’s a potential conflict of interest – like being an investor – I always disclose my interest. There is no such conflict here, I strive to be independent in my viewpoints. For more details, see my ethics statement.

1. Context and motivation

Khan Academy is a US-based non-profit education provider, which teaches students about math, art, computing and many other topics for free, with videos and interactive learning guides. Courses are aimed at middle school to high school students, and also college students. The site is one of the most popular free education sites, with millions of users.

Khan Academy stands out from many other non-profits not just for its strong funding; for example, grants from the Bill and Melinda Gates Foundation and other philanthropic organizations, but also its strong engineering team. John Resig, creator of JQuery, is its chief software architect.

Back in 2017, an idea to come up with a better architecture started being raised more often within the engineering team. Back then, Khan Academy’s backend consisted of a Python monolith exposing REST endpoints. It was during that year that the team started experimenting with GraphQL for their API. By the end of 2017, the team realized that GraphQL had many benefits and decided to deprecate the REST interface and migrate existing endpoints to GraphQL.

It was also around then that the team started to feel the pain of maintaining a monolith with Python, whose performance was not great and resulted in more hardware resources being needed to run the monolith.

Python 2’s end-of-life announcement was the final spur to start the project. In 2019, Python Software Foundation announced Python 2’s end of life date as January 2020. It was this which kicked off the rewrite project because it meant several goals needed to be achieved:

Get off Python 2 as soon as possible. With Python 2 reaching end of life, delay was not an option.
Replace the REST API with GraphQL.
Break the Python monolith into services.

2. Kickoff and phasing

The rewrite project got underway in June 2019. The team started by choosing which language to move over to; Python 3? Kotlin? Go? They considered all options and settled on Go, largely for its first-class support in the Google App Engine, and the simplicity and consistency of its language and performance. As Kevin Dangoor summarized:

“Moving to Kotlin was an appealing alternative. While we were at it, we decided to dig deeper into other options. Looking at the languages with first-class support in the Google App Engine, another serious contender appeared: Go. Kotlin is a very expressive language with an impressive set of features. Go, on the other hand, offers simplicity and consistency. The Go team is focused on making a language which helps teams reliably ship software over the long term.

As individuals writing code, we can iterate faster due to Go’s lightning quick compile times. Also, members of our team have years of experience and muscle memory built around many different editors. Go is better supported than Kotlin by a broad range of editors.

Finally, we ran a bunch of tests around performance and found that Go and Kotlin (on the Java Virtual Machine) perform similarly, with Kotlin being perhaps a few percent ahead. Go, however, uses a lot less memory, which means that it can scale down to smaller instances.

We still like Python, but the dramatic performance difference which Go brings us is too big to ignore, and we think we’ll be able to better support a system running on Go over the years. Moving to Go will undeniably be more effort than moving to Python 3, but the performance win alone makes it worth it.”

The scope of the project was very clear from the start. Migrate 100% of Python 2 code to Go, have only GraphQL endpoints, and use GraphQL federation. The biggest task was agreeing on architectural strategies.

The team ended up converging on a federated GraphQL hub. The main difference between a REST-based “API gateway” is that with REST, a request is often directed to a single service. With GraphQL gateways, a query plan is generated that includes data from multiple backend services.

The proposed architecture looked like this, at a high-level:

A high-level overview of the new architecture at Khan Academy, based on federated GraphQL. The actual number of services is a lot higher at around 40, but the architecture is very similar. Diagram source: Khan Academy’s engineering blog.

How do you estimate how long it takes to migrate 1 million lines of code? I asked Brian, who said they used pretty straightforward heuristics and refined it later.

“As far as estimates go, we started by estimating the project using heuristics. We estimated the lines of code the average developer would likely port per day. We then used that with the number of lines of Python we had, in order to figure out how much work each service would take.

This was a good first pass, but when we got started working on APIs in the services, we had to break it down further. Some things were a lot more complex than that heuristic could manage. So the estimates changed when we dissected the work into smaller parts.”

The project was split into two parts:

Phase 1: Minimum Viable Experience. It’s common to call the first phase an MVP (Minimum Viable Product,) referring to a barebones but usable milestone. However, the team did not want to call this phase an MVP, because they already had a product. MVE answered the question of what the key features were, which if removed, would materially alter Khan Academy’s identity. The MVE scope was mostly defined by product managers looking at the experience from a product point of view.

These MVE features were things like content publishing, content delivery, progress tracking, user management, and more.

This phase alone took about 2 years to complete, completing in August 2021. At this point, about 95% of traffic was flowing through the new code and 32 services were built.

Phase 2: endgame. The second phase was everything else. Although it might have seemed like the project was largely complete, with 95% of traffic going through the new services; in fact there was much to do. On top of rewriting the remainder of the Python monolith still serving traffic, all internal tools needed to be rebuilt from Python to Go, which was a sizable undertaking in itself.

This second phase took another 1.5 years to complete. Part of why it took so long was because Phase 2 was less staffed with engineers than with Phase 1 had been. Also, in this phase, new features were built, as well as existing ones being migrated. In this phase, another 5 new services were created and a few developer-only services were added, bringing the services count to just over 40.

A big part of the endgame phase was adding non-MVE behaviors to existing services.

Here’s some statistics on how each phase went:

3. The migration phase

The team wanted to avoid a “big bang” migration in which everything is rewritten at once. Instead, they chose a “field by field” approach. As Brian shared:

“We knew a ‘big bang’ rewrite would be fraught with pain, so we chose the opposite: not at the service level, the feature level, or the model level, but at the field level. Literally, we'd move User.FirstName over to the Go-based User Service while User.LastName was still in the Python Monolith. We did this with GraphQL federation and some side-by-side testing magic which we added to our federation service.”

Before doing any migrating, the team needed to build basic infrastructure for the new service. This meant putting the GraphQL federation service in place.

The first migrated service was the simplest; a service hosting a single field. This service was one which answered the question of what was the oldest version of the mobile client that could be supported. When this first service migration was complete, the GraphQL federation service was built, and one Go service was built which served real – if very low – levels of traffic, while the majority of traffic flowed through the existing monolith.

Khan Academy’s system after migrating the first and smallest service

The migration strategy applied a similar side-by-side testing approach to all services.

Step 1: optional shadowing of traffic on the new Go service. The Python result is returned. The Go service only gets the request if developers set a parameter to also direct traffic to their new service. If no parameter was set, then no traffic was sent to the new service. Playing with this parameter allowed engineers to test the new service as they built it, without any risks to the production system.
Step 2: side-by-side testing of GraphQL services. Both the Python and Go services are called and any differences in responses logged, and the Python service’s result is returned.
Step 2.5: canary. A configurable percentage of traffic starts to return responses from the Go service, while still in a side-by-side testing phase. While side-by-side testing can be used for GraphQL queries (fetching data); for GraphQL mutations (modifying server-side data,) it is not a good option. It was GraphQL mutation use cases when a canary was used. Learn more about GraphQL queries vs mutations.
Step 3: migration. Only the Go service gets the traffic, but the Python service is still present, in case it needs to return as the primary.
Step 4: remove the Python code and service. Migration complete!

The automated testing approach was interesting. With any big rewrite and migration, having a robust automated tests suite can be an additional safety net. At the same time, porting over the unit or integration test suite can also be lots of work. So how did Khan Academy proceed? Brian explains:

“Our unit test coverage in Python varied: some areas were well-covered, while others were not. When it made sense, we ported over our Python tests to Go, and used them to build out the Go behavior. That worked well for simple, stateless functions. But in reality, most of the code was structured differently in Go, so the python tests didn't help too much.

We wrote a LOT of tests in Go, however. Our test coverage is much better today because of it. We also took a more “behavioral” approach to testing. We wanted the APIs to be ported identically. In doing so, we knew that for any input in the Python and Go code, we'd want the Go code to be exactly the same as the Python implementation. This is where "side-by-side" testing came in as part of the migration.”

The fact the migration took a long time was tough on the team. I’ve experienced first-hand how draining year-long migrations can be, so I asked Brian how he and the team handled their migration, in which just the first phase took more than two years. Brian was candid. He shared that it was rough:

“Most people who come to work at a non-profit like Khan Academy do so for the mission, not for the technology. People want to push forward our feature sets, expose more content to our learners/teachers. The technology tends to be second.

Engineers saw this migration as an existential imperative that they could contribute to, so this viewpoint helped. We also had some REALLY interesting technical problems to solve during the migration and these challenges resonated with many of them. However, each and every one of us were ready to finish the migration after it had been going on for 3.5 years.

Such a long migration was more challenging to stomach from the wider organization's perspective. There were big time frames during this project during which engineering did not build any new features because the migration took up all our time! On one end, this gave product managers and designers time to make plans for what features we’d build in a post-Goliath Khan Academy. Still, during the migration, both Product and Design faced a lot of pushback from engineering, and unfortunately we saw a lot of attrition in these functions, perhaps also as a result of less progress in building new things during this time.

By the end of the project, all of us: engineering, product, design and the rest of the business, just wanted the migration to be over so we could move into the post-Goliath world. Now finally we’ve arrived and the renewed sense of energy and excitement is very clear. We did it, and we’re ready to get back to building!”

4. Things that worked well

The engineering team warmed to Go during the project. Khan Academy started the rewrite with around 5-10 engineers. As the MVE phase progressed, more engineers joined in and at the end of the MVE phase, the whole engineering organization of about 100 people worked on it. For the endgame, the project started with around 50 engineers and gradually ramped down to 4 engineers in the final days of the project.

Before the rewrite, few engineers had used Go in production, so it was interesting to hear the team’s impressions of it. Kevin Dangoor collected learnings two years into the project, after half a million lines had been migrated. He shared these learnings in a blog post:

Engineers liked Go. Some liked the ease of reading and writing, others praised the documentation, while the tooling and compiler speed all scored points with the team. A software engineer who came from the .NET world initially found it strange that Go doesn’t do exception-like error handling. Later, this engineer said of the fact that Go’s errors are values: “Being able to call a function that doesn’t return an error and know for sure that it must succeed, is really nice.”
Performance was excellent. Compared to Python, Go’s runtime performance was much faster. Talking with the team, I’m told they compared the service hour cost of operating the same code on Python and Go. The difference was an order of magnitude in favor of Go, which was up to x10 cheaper for certain types of requests.
The lack of generics was the biggest complaint. Generics refers to the ability to write a function that could work with a variety of data types and have the compiler automatically create versions for more specific types at compile time. This is a language feature that’s widespread in many modern languages like TypeScript, Java, C#, Python and Swift, which all support generics at some level. After years of planning, Go added generics in March 2022, in Go version 1.18. Google App Engine only added support for this version of Go in December last year, so the team has not yet embraced this feature, but will do soon!

Keep shipping incrementally. Kevin thought that setting the rhythm of shipping to be incremental but continuous was the difference between success and failure. The team very quickly shipped their first migrated service – even though it was a tiny one! – and then kept up the pace of always shipping and always migrating as well.

Kevin wrote about how taking small steps was harder at first but became easier as the project progressed:

“At the beginning of the project, we stressed the importance of working on small slices as much as possible. In those days, this was tricky because the goal might be to switch over one GraphQL field, but that field might depend on a variety of other machinery running inside the monolith. As time progressed, more of the other required parts had been ported, making further porting for the same service smoother.”

One service to “own” a piece of data. Kevin shared an early decision they took which made the migration much clearer: only one service would “own” a piece of data.

The team put a firm rule in place that only one service could write a given piece of data. All other services had to call this “owner” service via the API in order to make changes to the data. Kevin suggested that without this rule, figuring out how and why data changes happened would have been extremely difficult.

The side-by-side migration approach worked extremely well. When I asked Brian for an approach he’d use again for similar migrations, he selected the side-by-side migration approach as one. This was because the project felt like rebuilding an aeroplane while in mid-air.

With the side-by-side approach in place, the team got to scrutinize the differences and problems that showed up across thousands of individual fields. They got a very real sense of progress by inspecting the level of traffic their services received. Day by day, this traffic grew from 0% up to 100%.

One benefit of a side-by-side migration is that you can track the percentage of traffic served from the new system. Above is a rough illustration of the migration’s progress, as pieced together from talking with Brian and Kevin.

Treating this migration as a fixed scope, fixed timeline project was the right choice, in hindsight. When talking about the project management approach, Brian revealed something that initially surprised me: they did not follow an “agile” approach on this project like as usual:

“Although I have been a proponent of agile development practices for most of my career, we mostly treated this project migration as a waterfall one. The only "agile" bits had to do with borderless engineering, and how we prioritized the work. “Borderless engineering” is what we call the practice of individuals floating from team to team for short periods of time – a few days to a few months – to help out on work when it’s the sensible thing to do.

The migration was a fixed-scope, fixed-timeline project. We had a massive burndown chart that always gave us a good understanding of how we were doing. When a team fell behind, we moved engineers around. In the end, we finished 4 days before our fixed deadline (January 31, 2023.)

Looking back, treating this type of work as a fixed scope, fixed timeline project was exactly what we needed.”

My initial surprise derived from my experience of treating most projects as “agile”: build something quickly, get feedback, then reassess your plan and build whatever makes the most sense. This agile approach works well when you’re innovating or discovering a problem space. However, with this migration the problem space was a given and the scope of the work was well understood. So, it’s no wonder that sticking to the original plan worked well.

5. Choices worth reconsidering

I asked both Brian and Kevin which choices they might reconsider with the benefit of hindsight.

Switching to a brand-new language for the rewrite. Brian ran me through his reflections:

“Was switching to Go worth it in the end? Go is demonstrably faster than Python, therefore it is cheaper to run, which affects our bottom line.

However, don’t forget that nobody on the team knew Go in-depth when we started this project. So we had approximately 100 developers who all needed to ramp up on the technology. And, sure enough, we made mistakes along the way and those mistakes slowed us down.

Here’s something I don’t yet know: how long it will take for us to “reclaim” the cost of the ramp-up to Go: the time the team spent mastering this technology versus cheaper cloud costs. In hindsight, I would likely do an analysis of this tradeoff. Looking back, there is a possibility that Python 3 would have been more prudent for getting the project done, faster.”

The team played loose with the “port things exactly as they are” approach when it came to internal tools. Brian says:

“We also had a general rule for this project: we move the behavior over exactly. If there's a bug in the python code, we port the bug (most times.)

However, for our internal tools, we were looser with this rule. Instead of porting these tools one-on-one from Python to Go, we wrote new tools. This meant we couldn't easily use the side-by-side system. Building these new tools likely slowed us down compared to porting them. I'd revisit that as well.”

I’m glad Brian shared these learnings, as neither are ones that we engineers talk about much, even though we should. The reality of moving to a new language is that there will be a lot of time wasted on learning the new technology and on mistakes.

However, looking at the other side of the coin; this is time invested in engineers learning a new and interesting technology, and it’s motivating to work at places which support such investment. Also, companies that budget for engineers to invest in learning and using modern tools tend to have an easier time recruiting curious devs.

The same goes for internal tools. Sure, the prudent approach would have been to port the tools one-on-one. It would have been faster because little thinking is required.

But then again, what is the point of porting internal tools if they won’t be improved upon? This is just speculation, but it’s probable the migration was “dry” enough for engineers to channel their creativity and desire to play around with Go into internal tools, like many engineers do.

Both the decision on whether to change languages, and whether to stick strictly to the scope for developer tooling, are common challenges on many projects. The project might have been completed faster by being stricter about technology choices and sticking to the original scope (even with internal tools.) However, such strictness could have harmed the company’s engineering culture.

Would a stricter approach have resulted in a less fun and interesting place to work, and perhaps higher engineering attrition? These is a question I can’t answer, but as an engineering leader it’s worth considering, alongside the budget and timeline of a project.

6. Learnings from working on a 3 year-long project

Kevin Dangoor was part of the leadership team for the vast majority of the rewrite. Responding to my question about his learnings from it, he shared a few things.

Defining the “minimum viable experience” paid off handsomely. Even though there was no “minimum viable product” to discuss, defining what a “minimum” port looked like helped the team to focus and prioritize the right things.

Doing “as direct a port as possible” meant estimates slipped very little. At the kickoff of the project in summer 2019, the team estimated the MVE phase would take about 2 years, based on the number of fields to port and lines of code to move. In August 2021, the team completed the MVE, almost exactly as per their original estimate two years previously.

If you build software, you’ll know it’s hard enough to estimate for even a month ahead, so accurately doing this on a timescale of years was an impressive feat. Kevin said he thinks the estimate was accurate because they tried to do as direct a port as possible for the new system, not expanding its scope.

A project that “only” goes from monolith to microservices would likely have less complexity. Kevin said he thinks their migration added complexity in several ways:

Changing languages from Python to Go. For instance, this meant Python libraries could not be used in Go, so the team needed to find the best replacements they could.
Moving to new versions of Google Cloud APIs. The Google Cloud APIs evolved shortly before the migration started, and so moving to the newer Google Cloud APIs were part of it.
Moving to services meant splitting control of the data. In a monolith, all data was in one place. With lots of services, workflows became more complex and data flows changed.

A sense of dedication to finish the work and feeling pride when it was done. Kevin has been in the tech industry for 20+ years, at places like Mozilla, Adobe, and currently GitHub. Nonetheless, when I asked him how he feels about this project, he shared:

“The migration was a huge undertaking. I’m proud of having had a part to play in it. It was unlike anything I worked in my career, so far. Of course, I worked with great engineers and seeing their dedication to this project was very important.”

Brian took over leading Part 2 of the project, the Endgame phase. I asked him about his learnings. Here’s what he shared.

Hard deadlines can be motivational. Brian was working on the migration project at first as a software engineer. For the final phase, he took on the role of managing and coordinating the whole project. He shared an interesting observation about hard deadlines:

“It’s interesting to see how a hard deadline can be both motivating, and help people to organize around it. Our directive was clear: we will not slip in completing the migration by our date. If we need extra people, we'll get extra people.

Coordinating all the teams to get to that deadline was complicated; if they all finished their work by the deadline, the org wouldn't make the deadline because so many teams would be blocked by other teams. Having a hard deadline forced us to align in creative ways to ensure a ‘critical path.’ To make sure we finished in time, we had mini-deadlines for the last 6 months of the project. If X didn't happen by Y date, the project was at risk.”

Just because you have services, you cannot ignore the broader ecosystem. Another learning comes from Brian, that just because you move to services, you aren’t free to ignore the “monolith” or how the services work holistically. If you take a few steps back, the collection of services also appears as a “monolith,” although a loosely coupled one.

However, these services use shared resources. For example, if one service makes heavy use of Redis, other services that use Redis will be affected unless multiple instances of Redis are created to mitigate this, which is what Khan Academy ended up doing. Caching of data between co-dependent services can cause thundering herds when caches expire. "Smarter" cache expiry is necessary to maintain a system that can truly scale.

Takeaways

Many thanks to Brian and Kevin for candidly sharing their experience of a long and challenging migration. They’ve each been generous in giving their observations and I recommend following them on LinkedIn. In a comment below, Brian mentioned how Khan Academy is hiring for those who are interested in joining the company after this migration.

I’ve read plenty of engineering blog posts on teams celebrating the hitting of a milestone in a migration, or announcing a migration is done, which often are triumphant in tone. However, having been close to a multi-year migration at Uber, my experience is these projects certainly don’t feel like triumphs while they’re taking place!

Long-running migrations often feel thankless, never-ending and frustrating. Both Brian and Kevin were good sports in sharing the positives and I appreciate Brian being candid about how rough the project felt at times; not just for the engineering team, but also for product and design.

Even though Khan Academy has what feels like a strong engineering culture, Brian mentioned that people at the company “want to push forward our feature sets, expose more content to our learners/teachers; the technology tends to be second.”

In any company where most engineers are builders, migrations will feel like a drag. However, the more a company builds and the bigger it grows, so the need arises to migrate over to new systems. Although counter-intuitive, I suggest that to be a great product engineer, it’s worth familiarizing yourself on how to do migrations, so you can do them more efficiently and reliably.

Additionally, there’s fewer better ways to learn this than by working on and helping out with challenging migrations.

I hope this deep dive has painted a realistic picture of what a complex, multi-year migration looks like, and offer approaches that are useful, should you embark on a similarly challenging project.

BoxLang 1.00

Mike's Notes

"BoxLang is a modern dynamic JVM language that can be deployed on multiple platforms: all operating systems, web servers, Java application servers, AWS Lambda, ios, Android, Web Assembly, and more.

BoxLang combines many features from different programming languages, including Java, ColdFusion, Python, Ruby, Go, and PHP, to provide developers with a modern, fluent and expressive syntax. BoxLang has been designed to be a highly modular and dynamic language to take advantage of all the modern features of the JVM." - Ortus

BoxLang recently emerged from beta as version 1.0, which looks incredible. Since version 3, Pipi has been built using CFML code. Pipi 9 already uses some Ortus products as part of its stack. I have put complete migration to BoxLang on the roadmap for Pipi 10.

This means other languages can then be used in addition to CFML.

BoxLang is completely open-source on GitHub, and Ortus offers paid support. It requires JDK 21 or later.

Initially, Pipi will use the free open-source community-supported plan for BoxLang.

Once Ajabbi generates enough revenue, it will generously sponsor Ortus on an ongoing basis and purchase paid custom enterprise-level support from Ortus, including the development of extra modules, dedicated platform support, etc.

The team led by Luis Majano at Ortus are superb, and outsourcing makes sense.

Resources

References

Reference

Repository

Home > Ajabbi Research > Library > ColdFusion
Home > pipiWiki >

Last Updated

04/05/2025

Article

By: Ortus

BoxLang: 03/05/2025

BoxLang is a modern dynamic JVM language that can be deployed on multiple platforms: all operating systems, web servers, Java application servers, AWS lambda, iOS, android, web assembly, and more.

Dynamic Language

BoxLang is dynamically typed, meaning you don’t need to declare types if you don’t want to. It can do type inference, auto-casting and promotions between different types. The language adapts itself to its deployed runtime. It can add/remove/modify methods and properties at runtime. Making it highly flexible and adaptable.

Dynamically typed just like CFML, but we go further…
JDK21+ Minimum
JSR-223 compatible
No reflection, we use InvokeDynamic for everything
Type inference, auto-casting, type promotions
Interface and superclass default method promotions
Use all new JDK features and types

Multi-Runtime Development

BoxLang has been designed to run on multiple platforms. This allows you to write adaptive code for any Operating System JVM, a servlet container web server, cloud lambda functions, iOS, Android, or even the browser via our web assembly package. BoxLang® builds upon its language core so it can be deployed on almost any running platform, present or future.

Coming Soon:

Android
IOS
WebAssembly
Microsoft Azure Functions

Established Ecosystem

Even though BoxLang is a new language, it already has an established ecosystem, since every Java and ColdFusion/CFML library works already with BoxLang. This was our priority when designing BoxLang, that it would automatically be able to integrate and run libraries from the Java and ColdFusion/CFML ecosystems. It ships with CommandBox as it's package manager, server manager, task manager and REPL tool. Almost any project in https://central.sonatype.com and https://www.forgebox.io should work with BoxLang.

Java Interoperability

BoxLang is 100% interoperable with Java. You can extend and implement Java objects, use annotations, declare classes, imports, and more. Thanks to InvokeDynamic and our BoxLang DynamicObject core, everything in BoxLang is interoperable with Java.

If you are a Java Developer you will feel right at home, but with tons of more tools, functional programming and less ceremony.

Example Here

import java:ortus.boxlang.runtime.scopes.Key;
import java:java.util.concurrent.ConcurrentHashMap;

myMap = new ConcurrentHashMap();
result = Key.of( "hello" ).getName();

printLn( result );

Low Verbosity Syntax

BOXLANG is a low-verbosity syntax language. It is highly functional, fluent, and human-readable. Our intent with BOXLANG is to make it highly expressive and low ceremony.

Event Driven Language

BoxLang has an event bus internally that can be used to extend the capabilities of the language or even your own applications. You can listen to almost every part of the language, parser and even the runtime or collaborate with your own events.

Interceptors for the language
The best way to scale the language
Listen to the entire language life-cycle
Modules can even collaborate events

Example Here

function onScopeCreation( data f
    // Only listen to server scope
    if( data.scopeName.getName() == "server" ){
        // Add coldfusion struct
        data.scope.put
            Key.coldfusion,
            new ImmutableStruct i
                "InstallKit": ""
                "appserver": "JEE",
                "expiration": "",
                "productlevel" : "Enterprise",
                "productname" : "ColdFusion Server",
                "productversion": "2023,0,06,330617",
                "rootdir": "",
                "updatelevel" : "0"
            }）
        );
    }
}

Enterprise Caching

Powers internal caching, with custom providers, object stores, listeners, and stats. Inspired by CacheBox.

IDE + Tools

We provide not only the core language but also several tools to help developers do their job with ease and power. We provide a Visual Studio Code extension for the language to provide syntax highlighting, code debugging, code insight, code documentation, formatting, LSP integration, and much more. Our + subscribers get even more tools like enhanced debuggers, ColdFusion/CFML transformers, and much more.

BoxLang VSCode package:

Language Debugger
LSP
Executable scripts within the IDE
Converters: AST, CFML -> BoxLang, Bytecode and more
Visual Studio Image

CLI Tools

REPL
Runners
Converters: CFML to BoxLang, BoxLang to Bytecode

Scheduling & Task Framework

BoxLang's Scheduled Tasks Framework provides a centralized and portable way to define and manage scheduled tasks on your servers and applications. Source control your tasking with our scheduling DSL.

Inspired by ColdBox Tasks
Schedulers are portable, fluent and human
Task framework
Write them in Java or BoxLang
Access to any executor in Java
Run schedules as cron jobs, no server needed

Example Here

//* Declare the tasks for this scheduler */

@Override
public void configure() {
    task ("PeriodicalTask" )
	       .call () > System.out.println("++++++>>>>>> Hello from MyModuleTask" ) )
   	    .everySecond)
	       .onFailure( (task, exception) -> System.out.printin("MyModuleTask failed:" + exception.getMessage) ) )
	       .onSuccess( (task, result) -> System.out.printin( "MyModuleTask succeeded:" + result ) );

    task ( "OneOffTask" )
	       .call( ( System.out.printIn("++++++>>>>>> Hello from OneOffTask" ) );

    xtask( "A Disabled Task" )
	       .call( () -> System.out.printin( "Hello from A Disabled Task" ) )
	       .everySecond;
}

Functional Programming

BoxLang goes beyond traditional paradigms. Explore functional concepts like immutability and higher-order functions to write elegant and robust code.

BoxLang fosters code that's easier to maintain and less error-prone thanks to its support for functional programming principles. This translates to a more reliable and cost-effective development process.

Modular Extensibilty

BoxLang has been designed internally to support the concept of BoxLang modules that can enhance every aspect of the language or your applications built with BoxLang. BoxLang is one of the very first languages you can build upon using modules. You can add new built-in functions, templating components, new/modify functions on existing classes, functionality, Runtime Debugger, AOP aspects, or listen to events within the language itself.

Multi Parsers: CFML & BoxLang

BoxLang supports a dual parser and transpiler to execute ColdFusion/CFML code natively (maybe more languages later). This means that you can run all your ColdFusion applications within BoxLang natively. We also provide tooling to automatically transpile your ColdFusion code to BoxLang® as well if you have a + Subscription.

Our way to split with the old and bring in the new
Transpile CFML into BoxLang
BoxLang is a NEW clean slate
Future: Groovy to BoxLang

Professional Open Source

BoxLang is a professional open-source project based on the Apache 2 license. Ortus Solutions supports every aspect of the language, and you can get a BoxLang+ subscription to not only professional support but also extended features and modular capabilities.

BoxLang Community (Free & OS)

Apache

Two Commercial Subscriptions

BoxLang+
BoxLang++
Custom

Serverless Development

BoxLang empowers serverless development. Write code in functions that trigger based on events, and leave server management to the cloud provider. This means faster development cycles, effortless scaling, and cost-effective applications.

Meta-Programming

Metaprogramming in BoxLang is a powerful technique that allows developers to modify the behavior of BoxLang classes and objects at runtime. It involves manipulating the way classes and objects respond to method calls, property access, and more.

Quick Installer

You can also run this script to install BoxLang in your Windows, Mac or *nix.

Bash/ZSH

        /bin/bash -c "$(curl -fsSL
        https://downloads.ortussolutions.com/ortussolutions/boxlang/install-boxlang.sh)"

        /bin/sh -c "$(curl -fsSL
        https://downloads.ortussolutions.com/ortussolutions/boxlang/install-boxlang.sh)"

Windows PowerShell

        iex ((New-Object
        System.Net.WebClient).DownloadString('https://raw.githubusercontent.com/ortus-boxlang/boxlang-quick-installer/main/src/install-boxlang.ps1'))

Downloads

Runtime

BoxLang OS/CLI/JSR223

Tools

VSCode IDE

CommandBox

You can easily start up a server with CommandBox.

server start cfengine=boxlang javaVersion=openjdk21

Science, Promise and Peril in the Age of AI

Mike's Notes

Here is a special feature from Quanta Magazine, a series of AI articles.

Resources

https://www.quantamagazine.org/series/science-in-the-age-of-ai/

References

Reference

Repository

Home > Ajabbi Research > Library > Subscriptions > Quanta Magazine
Home > Handbook >

Last Updated

03/05/2025

Science, Promise and Peril in the Age of AI

By: Various authors

Quanta Magazine: 30/04/2025

It started as a fantasy, then a promise — inspired by biology and animated by the ideas of physicists — and grew to become a powerful research tool. Now artificial intelligence has evolved into something else: a junior colleague, a partner in creativity, an impressive if unreliable wish-granting genie. It has changed everything, from how we relate to data and truth, to how researchers devise experiments and mathematicians think about proofs. In this special series, we explore how AI is changing what it means to do science and math, and what it means to be a scientist.

Input

Where we are right now, and the science of how we got here.

Black Box

The innards of the most advanced AI systems are inscrutable, but perhaps not for long.

Output

An exploration of the profound impact the new technology is having on science and math.

Video

How to Build an "Artificial Scientist".

BoxLang vs the World by Kai Koenig

Mike's Notes

A video of the presentation by Kai Koenig at CFCamp 2024.

Resources

References

Reference

Repository

Home > Ajabbi Research > Library > ColdFusion
Home > Handbook >

Last Updated

04/05/2025

BoxLang vs the World by Kai Koenig

By: Kai Koenig

On a Sandy Beach: 27/06/2024

Kai Koenig of Ventego Creative is a "trained mathematician and went into software development after uni (and ignored my maths background for a long time).

With the advent of FP and machine learning/data science, I got interested again and found it reasonably easy to adopt these technologies with the help of having an understanding of the maths behind it.

Some of my friends didn't and we wondered why. We realised that the problem was not only the maths, but quite often also that they would struggle with finding ways to start (re-)learning some of it after having given up on math in high-school."

Kai Koenig's session comparing BoxLang to established languages like JavaScript, TypeScript, Kotlin, Scala, Ruby, and Java. He emphasized BoxLang's unique features and better tooling compared to existing CF platforms. - YouTube

AllPinouts

Mike's Notes

AllPinouts is a useful reference website I found while trying to connect some older servers to test Pipi 9.

Resources

References

Reference

Repository

Home > Ajabbi Research > Library >
Home > Handbook >

Last Updated

01/05/2025

AllPinouts

By: Nicola Asuni

AllPinouts: 20/04/2025

AllPinouts is a Web-based free content project to list cable and connectors pin-outs.

Pinout or pin-out is a term used in electronics to describe how an electrical cable is wired, or the function of each wire (pin) in a connector.

An electrical connector typically consists of several electrical contacts or pins that can be used to carry electrical power or signals. Due to the wide variety of applications and manufacturers, a wide selection of electrical connectors exists with different types and numbers of contacts or pins. The pinout of a connector identifies each individual pin, which is critical when creating cable assemblies and adapters. Proper identification of pins and wires ensures that the signals and power are transmitted across cables and connectors.

The archive is organized into the following categories:

Cables
Connectors

The Most Popular Deployment Strategies Explained.

Mike's Notes

Another great visual explanation from Nikki Siapno, who writes the weekly Level Up Coding newsletter. It's worth subscribing to.

Resources

References

Reference

Repository

Home > Ajabbi Research > Library > Subscriptions > Level Up Coding
Home > Handbook >

Last Updated

30/04/2025

The Most Popular Deployment Strategies Explained.

By: Nikki Siapno

Level Up Coding: 27/04/2025

Nikki Siapno, is an Engineering Manager at Canva and founder of Level Up Coding.

Using the right deployment strategy is crucial for seamlessly integrating new features and updates. It reduces the risks, avoids interruptions, and delivers a smooth user experience.

There are many possible approaches, let's look at 5 of the most popular deployment patterns:

Blue/Green Deployment

Renowned for zero downtime, this method uses two environments, Blue and Green. One hosts the live version while the other tests the new version.

After comprehensive testing without affecting live traffic, users are transitioned to the updated environment.

If an issue is discovered after switching environments, it is relatively easy to switch back.

The main challenge is the cost and complexity of managing two environments.

Canary Deployment

Named after canary birds in mines, it starts by rolling out changes to a small subset of users.

This allows for monitoring performance and gathering feedback.

If successful, you gradually extend the update to more users.

It excels in minimizing user impact during updates due to isolation of a small set of users.

Rolling Deployment

Updates software in phases, rather than all at once.

It incrementally upgrades different segments of the system, ensuring most of it remains operational during the deployment.

It can be ideal for critical systems that require continuous operation.

However, it extends the total update time and might introduce temporary inconsistencies.

Feature Toggles

Think of feature toggles as on-off switches for new features.

They allow teams to deploy features quietly, turning them on for specific users when it makes sense.

Feature toggles support strategies like canary releases and A/B testing.

The challenge lies in managing numerous toggles, which can become complex and risk feature conflicts.

A/B Testing

Comparable to a scientific experiment, A/B testing offers two variations of a feature to different user groups to gauge which performs better.

It's a go-to for validating user preference and effectiveness of new features, based on concrete data like user engagement or ease of use.

Each deployment pattern stands out for specific strengths:

Blue/Green for safety and zero downtime.
Canary for controlled, low-risk rollouts.
Rolling for maintaining continuous operations.
Feature Toggles for flexible feature management.
A/B Testing for data-driven user insights.

The right deployment strategy varies depending on the project's needs and objectives. Recognizing these differences allows teams to choose the best approach for a successful and user-centric software release.

The Python SQL Toolkit and Object Relational Mapper

Mike's Notes

I discovered SQL Alchemy today. I copied these notes from different pages of that website to remind me.

Resources

References

Reference

Repository

Home > Ajabbi Research > Library >
Home > Handbook >

Last Updated

29/04/2025

The Python SQL Toolkit and Object Relational Mapper

By: Mike Peters

On a Sandy Beach: 29/04/2025

Mike is the inventor and architect of Pipi and the founder of Ajabbi.

"SQLAlchemy is the Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL.

It provides a full suite of well known enterprise-level persistence patterns, designed for efficient and high-performing database access, adapted into a simple and Pythonic domain language." - SQL Alchemy

Dialects

The dialect is the system SQLAlchemy uses to communicate with various types of DBAPI implementations and databases. The sections that follow contain reference documentation and notes specific to the usage of each backend, as well as notes for the various DBAPIs.

All dialects require that an appropriate DBAPI driver is installed.

Included Dialects

PostgreSQL
MySQL
SQLite
Oracle
Microsoft SQL Server

Included, but not currently supported dialects

The following dialects have implementations within SQLAlchemy, but they are not part of continuous integration testing nor are they actively developed. These dialects may be removed in future major releases.

Firebird
Sybase

External Dialects

Currently maintained external dialect projects for SQLAlchemy include:

Database	Dialect
Amazon Redshift (via psycopg2)	sqlalchemy-redshift
Apache Drill	sqlalchemy-drill
Apache Druid	pydruid
Apache Hive and Presto	PyHive
Apache Solr	sqlalchemy-solr
CockroachDB	sqlalchemy-cockroachdb
CrateDB	crate-python
EXASolution	sqlalchemy_exasol
Elasticsearch (readonly)	elasticsearch-dbapi
Firebird	sqlalchemy-firebird
Google BigQuery	pybigquery
Google Sheets	gsheets
IBM DB2 and Informix	ibm-db-sa
IBM Netezza Performance Server	nzalchemy
Microsoft Access (via pyodbc)	sqlalchemy-access
Microsoft SQL Server (via python-tds)	sqlalchemy-tds
Microsoft SQL Server (via turbodbc)	sqlalchemy-turbodbc
MonetDB	sqlalchemy-monetdb
SAP Hana	sqlalchemy-hana
SAP Sybase SQL Anywhere	sqlalchemy-sqlany
Snowflake	snowflake-sqlalchemy
Teradata Vantage	teradatasqlalchemy

8 Common Mistakes to Avoid in API Design

Mike's Notes

Another great visual explanation from Nikki Siapno, who writes the weekly Level Up Coding newsletter. It's worth subscribing to.

Resources

References

Reference

Repository

Home > Ajabbi Research > Library > Subscriptions > Level Up Coding
Home > Handbook >

Last Updated

28/04/2025

8 Common Mistakes to Avoid in API Design

By: Nikki Siapno

Level Up Coding: 27/04/2025

Nikki Siapno, is an Engineering Manager at Canva and founder of Level Up Coding.

Clean, reliable APIs are a hallmark of strong engineering teams.

They don't happen by accident — they happen by design.

Let's walk through 8 critical mistakes to watch for when crafting APIs that stand the test of time.

1) Not Validating User Inputs

Input validation is your API’s first line of defense.

Accepting unvalidated data leaves you vulnerable to injection attacks, corrupted databases, and confusing downstream errors. Always validate inputs on the server side; not just for type and format, but also for business logic. Assume nothing about client behavior.

Tip: Use schema validation libraries or frameworks to enforce strict rules at your API boundaries.

2) Weak Security

Security can't be an afterthought. Common oversights include exposing sensitive data, improperly handling authentication tokens, or missing encryption on data in transit. A single weak endpoint can compromise your entire system.

Tip: Apply principles like least privilege, secure authentication (OAuth 2.0, OpenID Connect), encrypted communication (TLS), and regular security audits across your API surface.

3) Overcomplicating the API

A powerful API isn’t necessarily a complex one.

When you overload endpoints with too many responsibilities, mix resources and actions, or create deeply nested payloads, you make your API harder to learn, harder to use, and harder to maintain.

Tip: Stick to clear, predictable patterns. One resource → one endpoint. One action → one HTTP method.

4) Lack of Clear Documentation

The best API in the world won't gain adoption without great docs.

Unclear documentation forces developers to guess how your API works — leading to mistakes, frustration, and support tickets.

Tip: Invest in easy-to-navigate documentation, examples, and usage guides. Tools like Swagger/OpenAPI and Postman can help you automatically generate high-quality docs from your API definitions.

5) Poor Error Handling

Errors are inevitable — unclear errors are unforgivable.

Vague 500 errors, inconsistent error formats, and missing error codes make debugging a nightmare for users.

Tip: Standardize your error responses. Include meaningful HTTP status codes, error messages, and even error IDs that developers can reference for troubleshooting.

6) Not Rate Limiting

Without rate limiting, a single client can unintentionally (or maliciously) overwhelm your API and bring your system down. This isn’t just a problem for high-traffic APIs — any public-facing service needs basic protections against spikes.

Tip: Implement sensible rate limits (eg; 1,000 requests/minute per API key) and return clear error messages (HTTP 429) when limits are exceeded.

7) Not Implementing Versioning

APIs evolve, but your users shouldn’t suffer every time you make a change. Without versioning, even minor updates can break client applications that depend on your previous behavior.

Tip: Plan for API versioning from day one. Common strategies include path versioning (/v1/users) or header-based versioning (Accept: application/vnd.company.v1+json).

8) Ignoring Performance Optimization

An API might work fine at low scale, but poor performance can quietly become a huge bottleneck as adoption grows. Inefficient queries, redundant data transfers, and lack of caching can grind systems to a halt.

Tip: Monitor and optimize your most critical endpoints. Techniques like pagination, compression, selective fields (fields=id,name), and caching (ETags, HTTP caching) make a big difference.

Final Thoughts

Your API is a product — even if it’s "internal." Treat it like one.

Clear contracts, secure practices, thoughtful UX, and strong documentation aren’t just "nice to have"; they determine whether developers love integrating with your systems or dread it.

Every decision you make at the design stage pays compounding dividends later. Start strong, and you’ll save yourself (and your users) a lot of pain.

Data Centre options for Pipi 9

Mike's Notes

I'm writing down my evolving thoughts about a future Pipi 9 data centre. This is to enable suppliers, Alex, and others to give me feedback.

Update 01/04/2026

Anthropic provides each customer with a MacBook Mini to create an isolated cloud environment. Pipi could easily do something similar using low-cost refurbished PC's in the data centre. I'm running experiments with a Dell OptiPlex 7050 to see if it can host a single Pipi core for a customer.

Resources

References

Reference

Repository

Home > Ajabbi Research > Library >
Home > Handbook > Company > Values

Last Updated

01/04/2026

Data Centre options for Pipi 9

By: Mike Peters

On a Sandy Beach: 26/04/2025

Mike is the inventor and architect of Pipi and the founder of Ajabbi.

Pipi 4 in 2007

Pipi 4 used three ModemPak 45u rack cabinets to mount the servers. There was a separate 6 kw UPS. Here is a grainy photo from 2007, when it hosted the 17th most popular website in NZ, which received 30,000 unique visitors and 750,000 page views per month. It included servers for ESRI GIS, databases, applications, web hosting, updates, backups, security, rendering, and more, with extensive redundancy. It was never hacked due to the platform team led by Dave Evans. A report at the time to the Department of Conservation (DOC), written by Turing Solutions, a New Zealand government contractor, stated that Pipi 4 was worth NZ$3 million.

The first Pipi 9 rack cabinet

I am building out the first 45u rack cabinet. It was previously used for Pipi 4 and was in storage for 10 years. I'm having to replace the keyed barrel locks and get some rackstuds.

This temporary rack will be equipped with older servers for testing Pipi 9 and for rendering open-source modular applications to be sent to GitHub. Older servers work fine, but the rendering takes longer.

I will then replace these servers over the coming months with better second-hand ones to speed up the rendering.

I'm really enjoying this job. If you've ever had Mechano, you'll understand. I also have to design and build some specialist electronic gear to put in the rack. Even more fun.

It's also a significant milestone reached in the Pipi 9 build. This will be sufficient till Ajabbi scales.

The first Data Centre

Once scaling begins, Pipi 9 will need a data centre to serve as a render farm, automatically creating customised SaaS enterprise applications based on user requirements. The data centre will be completely isolated from the internet to maximise security. It can be expanded in stages if it is planned appropriately.

Each industry and each enterprise customer will get a dedicated server to store a mirrored copy of their deployment configuration and parameters, including localisation. No user data will be stored.

This enables customised updates to be created. For example, importing the latest version of Snomed or IATA.

Colocated Staging Data Centre

The built application then needs to be copied to disk and deployed to a small, caged, colocation data centre with high security, reliability, and connectivity, serving as a staging point for worldwide deployment. One option could be to use the proposed DataGrid Data Centre, set to open in Southland, NZ, by 2028, which will be connected via high-speed undersea cables.

Pipi 9 is designed to function as a swarm, with complete self-managing automation at every stage.

The emphasis is on 100% reliability and security, with extensive redundancy. See the Ajabbi Handbook reference for more initial information.

Production in the Cloud

The built application must then be deployed to a customer-selected cloud provider so the customer can use it in production. Pipi 9 would provide rapid updates and system changes within hours, not weeks or months.

Data Centre equipment

I really like the wide range of cabinet options that Rittal provides. My preference would be to use a standard 800mm wide 19 rack that could be flexibly used for;

Power
Cooling
Servers
Networking
Rack automation and monitoring

Other issues;

Determine the correct cabinet model to use
Use Open19 rather than OPC.
Use a cold aisle
Security doors
24 hr monitored access

Looking good

I also like the look of the Eaton racks, probably because I like black racks. But maybe Rittal could provide them in black as well, on special order from their manufacturing facility across the ditch in Australia. Or else I could have them spray-painted and oven-baked locally. The Data Centre needs to be cool and look cool, and I don't like messy network cabling. Hey, I'm an artist too.

No doubt these ideas will evolve.

Rack Shelving

Shelving from Rack Solutions is designed to hold micro-format PCs.

An example is this 5U HyperShelf for 16 Dell OptiPlex Micros, which costs US $599 for the kit.

Pages

Mike's Notes

Resources

References

Repository

Last Updated

Real-world Engineering Challenges #8: Breaking up a Monolith

1. Context and motivation

2. Kickoff and phasing

4. Things that worked well

5. Choices worth reconsidering

6. Learnings from working on a 3 year-long project

Takeaways

Further reading

Mike's Notes

Resources

References

Repository

Last Updated

Article

Dynamic Language

Multi-Runtime Development

Established Ecosystem

Java Interoperability

Low Verbosity Syntax

Event Driven Language

Enterprise Caching

IDE + Tools

Scheduling & Task Framework

Functional Programming

Modular Extensibilty

Multi Parsers: CFML & BoxLang

Professional Open Source

Serverless Development

Meta-Programming

Quick Installer

Downloads

CommandBox

Mike's Notes

Resources

References

Repository

Last Updated

Science, Promise and Peril in the Age of AI

Input

Black Box

Output

Video

Mike's Notes

Resources

References

Repository

Last Updated

BoxLang vs the World by Kai Koenig

Mike's Notes

Resources

References

Repository

Last Updated

AllPinouts

Mike's Notes

Resources

References

Repository

Last Updated

The Most Popular Deployment Strategies Explained.

Blue/Green Deployment

Canary Deployment

Rolling Deployment

Feature Toggles

A/B Testing

Mike's Notes

Resources

References

Repository

Last Updated

The Python SQL Toolkit and Object Relational Mapper

Dialects

Included Dialects

Included, but not currently supported dialects