Why Is Everything So Slow In Large Companies?

Mike's Notes

Recognition of the scale of a significant problem that exists in large enterprise systems.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library > Subscriptions > Smashing Magazine
  • Home > Handbook > 

Last Updated

11/06/2025

Why Is Everything So Slow In Large Companies?

By: Vitaly Friedman
LinkedIn: 06/06/2025

If you work in a large organization, you might find yourself puzzled by how slow and seemingly inefficient they can be. Decisions take time. Reviews tend to run in circles. Meetings always start late and overrun. Features get infinitely delayed. And teams become growingly protective of their silos.

And with larger companies, it compounds dramatically to the point that shipping on time becomes an exception, rather than the rule. But why does that happen? Why do projects have to be so painfully slow to have the slightest chance of being successful? And how do move the needle in the right direction? Well, let’s get to the bottom of it.

Heads up: Meet How To Measure UX and Design Impact  — on how to show the business impact of your incredible UX work. Use a friendly code LINKEDIN  to save 15%. Jump to table of contents.

“Wicked Problems”

At best, slowdowns are highly inefficient and wasteful. And at worst, they create a poor culture that propagates throughout the entire organization. The result: teams that feel utterly frustrated, confused, slowed down or simply ignored. That’s when people stop talking in team calls and send their AI notetakers instead.

As Sean Goedecke beautifully explains, the reasons for that aren’t inefficient processes, poor coordination, or lack of competency. It’s simply a different scale of problems that need to be solved, with many dependencies, decisions, and business goals to meet.

Wicked problems are unique and interconnected, which means failures have big consequences on the business.

Large companies are heavily constrained by a very small but very consequential set of problems, called ”wicked problems”. These are deeply inter-connected problems that interfere with many features, actors, systems, flows, and users — and often live at the very heart of the organization.

These are typical challenges of wicked problems:

  • Problems are “unique” and not properly understood
  • Many dependencies, with legacy or third parties
  • Multiple stakeholders with conflicting agendas
  • Involve many different parts of an organization
  • Every solution affects parts of the entire system
  • Solutions aren’t right or wrong, but better or worse
  • Take a lot of time to evaluate and make decisions
  • Can never be properly solved, just addressed
  • Failures have big consequences on the business

If you find yourself in the middle of a wicked problem, you’ll need to shift gears. The only option to avoid disastrous failures is to slow down. You have to spend more time in planning and risk management before designing a single pixel on the screen. Every UX decision comes at a cost, and there will be people keeping a very close eye on these costs.

With wicked problems, you will always be perceived as a disruptor who endangers business-critical processes if you aren't meticulous and careful enough. As a result, you become more strategic about your UX work. It means assessing risks and setting up a testing strategy early, but also building working groups and design guilds. It also means running cross-team workshops to uncover dependencies, conflicts, and constraints early.

The Curse of Slow Shipping

As a company and its products keep growing, by default it becomes more difficult to ship new features. Each new feature will in some way interact with existing features and systems, critical user flows, potentially legacy systems, third-party vendors, and custom systems done by other units.

And as Dave Stewart noted, the "work" is never just the "work". It's meetings, reviews, research, experimentation, scoping, setup, infrastructure, procurement, iteration, maintenance, tooling, changes, omissions, nice-to-haves, scope creep, surprises, contingency, sudden change of timelines and priorities, updates and fixes. And with larger companies, it compounds dramatically to the point that shipping on time becomes an exception, rather than the rule.

In many companies, projects get indefinitely delayed, moved, cancelled or abandoned all the time. The sad reality is that despite all the incredibly hard work put in by UX and engineering teams, such projects are perceived at best as wasted efforts, and at worst as costly failures by “undeperforming” teams.

In fact, even if the work was completed well ahead of the schedule, the underwhelming impact of that work might still reflect badly on the entire team. The whole project might be perceived as a good idea poorly implemented — with little room for debates about small wins and successes here and there. And for that reason, I spend an enormous amount of prep work to filter out, prioritize, scrutinize and pre-test design ideas before heading straight into the design mode.

In complex projects, execution is at best around 35% of all work, with only 20% of time needed for planned work. That's why many estimates are utterly wrong. 

Ensuring that a feature can be neatly integrated with all dependencies around it takes time and effort — and the more complex the product, the more time and effort it will require with every single change. It holds true especially if a particular change is the main “hub” for key user flows or business priorities.

In larger products, failures can have disastrous consequences at scale. So every change must be meticulously reviewed. High-risk scenarios must be thoroughly addressed and mitigated. More planning and prep work is required as it’s the only way to reduce the likelihood of things going terribly wrong. The illustration by John Cutler below beautifully shows how the usual workflow plays out if not enough strategic thinking has been done.


Slowdowns are often caused by habits and structures deeply rooted in an organization. Addressing them is crucial to improving workflows. (Credit: John Cutler)

Plus, user flows typically need to be revised as features must be easy to find, easy to use efficiently but also difficult to make mistakes with. Not to mention the usual politics and powerplay as different units thrive for attribution, higher visibility, influence, and budgets for the upcoming year.

Unsurprisingly, with all of it in play, things slow down enormously as the project is being pulled and pushed and reshuffled and revisited over and over again.

How To Slowly Introduce Change

Most companies claim to put quality over schedule any time, but very often there is nothing more critical for companies than to ship frequently, and on time. In fact, research and UX work often are perceived as blockers of shipping early and refining on the go.

Im practice though, we need to know at least enough to not be wasting time on a feature that provides little business value or little user value. Research isn't a blocker but a filter for shipping projects that matter. And that's one of the most common points I tend to raise early.


Company Culture Playbook (Notion docs) is a fantastic effort to bring free tools, templates and resources to improve company culture (Image links to the Notion hub).

Slowness doesn’t mean that it’s impossible to make a change in such environments. But you will need enough patience, enthusiasm and trust to slowly start moving the needle in the right direction. Personally, I would start by zooming in on things that affect everyone in the team. Well-known bottlenecks that slow people down. Meetings that end without action points or run late. Communication channels with key decisions made all over the place.

Little changes there can make a huge impact on everyone, and people will notice. The goal is to help other people see the benefits that your contributions bring and build up confidence for your work and your good intentions. Sometimes that might be just enough to get them on your side to address wicked problems with the due diligence and attention they deserve.

The Little Book on Strategy, a wonderful little cheat sheet with actionable advice on strategy and leadership, by Peter Biehr (image linked).

Most importantly: become more strategic and calibrate expectations. We don’t know how our stakeholders work, so we shouldn’t expect that they know and understand design process. The more sincere and vulnerable you are, the more likely you are to get understanding and support, rather than fast turnaround requests.

Useful Resources

  • Company Culture Playbook, by OpenOrg
  • New Ways Of Working: Playbook For Modern Teams” (Notion), by Mark Eddleston 
  • The Little Book of Strategy, by Peter Bihr 
  • Design Is Taking Too Long. When Can We Ship?, by Pavel Samsonov 
  • How I ship projects at big tech companies, by Sean Goedecke

Happy Birds: How To Measure UX (Video + Live UX Training)

I've been spending quite a bit of time reviewing and drafting new sections for the video courses on UX:

  • Measure UX and Design Impact (8h + live UX training)
  • Smart Interface Design Patterns (15h + live UX training)
  • Both video courses come with a live UX training with 1:1 feedback and UX certification.
  • Use the coupon code 🎟 LINKEDIN to save 15 off.

Thank you so much for your support, everyone — and happy designing!

From Complexity to Clarity: How Natural Language is Transforming Software—and the Roles Around It

Mike's Notes

What Kingsley thinks the problem and solution are.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

10/06/2025

From Complexity to Clarity: How Natural Language is Transforming Software—and the Roles Around It

By: Kingsley Uyi Idehen
LinkedIn: 07/06/2025

Kingsley Uyi Idehen is founder & CEO at OpenLink Software | Driving GenAI-Based AI Agents | Harmonizing Disparate Data Spaces (Databases, Knowledge Bases/Graphs, and File System Documents)

Over the last thirty years, the software industry has become more powerful and pervasive—but also more complex. That complexity has largely stemmed from a persistent gap in user interface and user experience design. In response, a host of specialist roles emerged—systems integrators, support engineers, onboarding teams, and more—whose primary job was to help users cope with software’s friction.

Now, we’re at a watershed moment. Large Language Models (LLMs) and generative AI have introduced a long-missing component into the computing stack: natural language as a UI/UX primitive. This isn’t a minor improvement. It’s a tectonic shift.

Natural Language as a UI/UX Layer

Natural language radically reduces the barriers to software use. Complex interfaces, scripting, and even command-line knowledge can be replaced by simple conversation. In plain terms:

  • Installation? Simpler.
  • Usage? Smoother.
  • Support? Increasingly self-service.

We’re finally seeing a reversal in the historic pattern of humans learning machine syntax. Now, machines are learning ours.

But Beware: Trust Is Not a Feature

Despite the ease-of-use revolution, LLMs are not to be blindly trusted. They are not deterministic systems and not reliable sources of truth. They are language prediction models—powerful, yes—but still prone to hallucination, bias, and inconsistency.

This introduces a non-negotiable operational principle for this new AI-powered stack:

Never trust. Always verify.

This is not optional. It’s structural. And ignoring it creates massive risk.

Verification: The Next Critical Role

This is where things take a hopeful turn. Just as previous computing shifts created entire job categories—from spreadsheet auditors to database admins—the AI era is creating demand for Verifiers.

These are professionals focused on validating, guiding, and grounding LLM outputs within organizational and ethical boundaries:

  • Prompt designers and safety verifiers who shape input for clarity and reduce harmful or misleading outputs.
  • Knowledge graph curators and fact-checkers who ensure that model outputs are grounded in trusted data.
  • Human-in-the-loop reviewers who act as decision and ethics buffers for AI-influenced operations.
  • AI UX designers and workflow overseers who ensure that natural language interfaces are both useful and safe.

This isn’t about job loss—it’s about job evolution. Manual support and integration roles may fade, but in their place we’ll see a rise in oversight, context-building, and orchestration roles.

Historical Perspective: Every Abstraction Brings Risk

The history of computing is the history of abstraction:

  • From binary to assembly.
  • From terminals to graphical interfaces.
  • From scripting languages to automation platforms.
  • And now, from structured commands to natural language dialogue.

Each step has made computing more accessible—and each has come with new vulnerabilities, new dependencies, and new responsibilities.

AI is no different. In fact, it may be the most powerful—and most dangerous—abstraction yet.

If we fail to adapt, if we delegate blindly, or if we stagnate in legacy thinking, this shift could tip the balance of control in ways we’re unprepared to manage.

Adapt Early. Verify Always. Protect the Future.

This is not just a technical evolution. It’s a societal one. And those who move early—who learn how to harness LLMs, verify outputs, and embed safety and trust into their AI systems—won’t just thrive. They’ll help safeguard the rest of us.

This is the work now:

  • To embrace the power of AI, without surrendering to it.
  • To build new tools, and new roles, that ensure trust is earned—not assumed.
  • To balance innovation with accountability.
  • To create software that’s not only easier to use, but also safer, more transparent, and more human-centric.

Some initial notes on what my first customer taught me

Mike's Notes

The first in a series of notes reflecting on what my first customer has taught me so far. 

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

12/06/2025

Some initial notes on what my first customer taught me

By: Mike Peters
On a Sandy Beach: 09/06/2025

Mike is the inventor and architect of Pipi and the founder of Ajabbi.

Mr G, my excellent former Startup Aotearoa coach, had suggested that for my first customer, I find a "teaching customer" who was happy to be a crash-test dummy.

It was intended to take something that works in the lab and apply it in the real world to solve real problems for a real customer, who would provide feedback, and I would learn some valuable insights.

What happened wasn't what I expected, and there were lots of surprises. I'm still learning something new every day by having to get out of the door and solve real-world problems (Thank you, Steve Blank).

First customer

This week, the website of my first (pro bono) customer went live as an early beta. It was a great success, despite there being a lot more to be done. 

The website is for a national disability organisation in New Zealand.

Requirements

The website should be published in different screen formats and languages, similar to how Wikipedia operates.

  • Screen formats
    • Mobile
    • Desktop
    • Braille device
  • Languages (number of speakers in NZ)
    • English (4,482,000)
    • Māori (213,000)
    • NZ Sign Language (20,000)
    • AAC picture language for non-verbal people, similar to what Prof. Stephen Hawking used. (?)

The login area should be easily customised to meet the specific needs of each registered user. Examples include

  • Colour-blind (14 types)
  • Autism
  • Dyslexia
  • Muscular Dystrophy (large buttons)
  • Deafblind (some of whom can't use screen readers, so use tactile braille)
  • Epilepsy
  • etc

Pipi CMS needs to provide extensive automation, making it easier for a volunteer team to handle the configuration aspects and allowing people with disabilities to maintain the website in the long term.

Many disabled people in New Zealand are isolated and impoverished because of the lack of government support. It needs to work for elderly people who often use much older computers, as well as for the young who use the latest technology. No one gets to be left behind.

It needed to be

  • simple
  • predictable
  • reliable
  • work on any screen or device
  • and assume nothing

The website needed to have the capacity to scale without restriction.

What happened

Pipi CMS can easily handle the work. The challenge was figuring out what it needed to produce. Designing and building this website had to be fast, cheap, simple and able to be done by volunteers who also faced their own health and support issues.

No complex tools, such as Figma, were used.

A great deal of rapid experimentation was conducted to determine what would work, using straightforward technology.

  • Built the 1200-page website by hand
  • Used simple standard layout templates
  • Used real data
  • Performed extensive search and replace operations
  • HTML Pages were manually laid out using an outdated but fast and straightforward method: hidden nested tables.

During this time, Google Search was blocked.

Testing

The beta version of the website was made available to many people with disabilities, their families, and support workers to gather feedback. Further modifications were then made rapidly until everyone was happy with the result.

Google Search was then unblocked, and a wider group was invited to use it.

Next steps

The existing layout templates (using nested tables) will be imported into Pipi CMS, connected to the content database and then automatically created by Pipi 9 CMS as static HTML pages most days.

All of the HTML code in these templates will then be converted to use CSS style sheets, which will make the website mobile-friendly (it's not yet). This will also enable significant changes and enhancements to the website's appearance.

A separate website for mobile with precisely the duplicate content, similar to what Wikipedia has done, will then be created.

The website will then be promoted to a broader audience.

Screen Reader

A significant effort will then be made to ensure it works seamlessly with screen readers, braille machines, and meets the WAI-ARIA 1.2 and WCAG 2.2 standards for web accessibility.

What I learned

Initially, attempting to make the website WAI-ARIA compliant proved to be a significant mistake, resulting in a 4-month delay. So was using style sheets.

It was much faster to create cheap and dirty HTML web page prototypes with rapid feedback loops measured in hours. The design and testing process went through several hundred cycles.

The biggest challenge was discovering the customer solution (how the website content and navigation were organised), not the technical means to deliver it.

How forgiving people were in an honest and open process, which initially went down many rabbit holes.

It was a privilege and an honour to be part of this. This kind of learning is priceless.

Chaos in the machine: How foundation models can make accurate predictions in time-series data

Mike's Notes

The GitHub link contains the experiment. Yuanzhao has done research into reservoir computing, a future enhancement for Pipi 11.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library > Subscriptions > SFI News
  • Home > Handbook > 

Last Updated

08/06/2025

Chaos in the machine: How foundation models can make accurate predictions in time-series data

By: Santa Fe Institute
Santa Fe Institute: 19/05/2025

Yuanzhao Zhang, Complexity Postdoctoral Fellow, Omidyar Fellow, Santa Fe Institute

Yuanzhao was born and raised in a small coastal city in China. His research focuses on collective dynamics on complex networks. In particular, he is interested in how individual differences shape collective behaviors and how complex dynamical patterns emerge from decentralized interactions.

Yuanzhao received a B.Sc. in Mathematics from Zhejiang University in 2014, an M.Sc. in Applied Mathematics in 2015 and his Ph.D.in Physics from Northwestern University.

Until recently, using machine learning for a specific task meant training the system on vast amounts of relevant data. The same was true for data representing a system that changes over time, says SFI Complexity Postdoctoral Fellow Yuanzhao Zhang. “The traditional paradigm in forecasting dynamical systems has always been that you need to train on the system you want to predict,” he says. If you want to forecast the weather in Santa Fe, start by training your model on the area’s historical weather data. 

But the advent of foundation models — a term coined in 2021 to describe the architecture at the heart of today’s AI systems — has changed the game. These models, like previous systems, train on large datasets. But unlike earlier, specialized deep-learning models, they’re designed to carry out a wide range of tasks. “They work right out of the box,” Zhang says. Notably, they can complete new tasks that weren’t included in their training data. For large language models, those include tasks like generating computer code or translating between languages. Reports of this behavior, called “zero-shot learning,” ignited a global race to build models that can similarly make zero-shot predictions for time-series data.

Zhang wanted to understand whether existing foundation models could predict chaotic systems and, if so, how they do it. In a recent analysis, Zhang and William Gilpin, a physicist at the University of Texas at Austin, reported that a foundation model called Chronos could generate predictions of chaotic dynamical systems at least as accurately as models trained on relevant data. Their paper was accepted to the Thirteenth International Conference on Learning Representations, which focuses on deep learning approaches in AI and was held in Singapore in April 2025.

Zhang says the paper represents the first test of zero-shot learning in forecasting chaotic systems, such as the weather and financial markets, which are governed by mathematical equations and extremely sensitive to small changes in initial conditions. Zhang and Gilpin tested their idea by using Chronos to predict how 135 chaotic systems would change over time. They tested each system using 20 distinct initial conditions. They compared the short- and long-term predictions of the model to deep learning models specifically trained using chaotic data. 

“We wanted to compare this zero-shot paradigm with the old paradigm and see if the foundation model can outperform the traditional models,” Zhang says. 

The promising results show that foundation models can make accurate predictions after training on data from any time series — not just data from the system or task that a user wants to predict. Forecasting the weather in Santa Fe may not require historical data, just other time-series behaviors in which the model could identify patterns. 

The study raises interesting ideas about what kind of training is required to accurately perform time-series tasks. “There’s this question: Do you actually need to learn chaos to have a good forecasting performance for chaotic systems?” Zhang asks. “I think the answer is no.” 

Zhang and Gilpin’s current work only looks at one-dimensional data; in future work, Zhang says he hopes to expand that to more complicated, multidimensional data. He’d also like to determine how the system carries out these tasks. “Is it, in some sense, learning the dynamics?” he asks. “Is it using anything more sophisticated than parroting?” 

The new study offers a step forward in answering those larger, deeper questions, he says. 

Streamline Your CI/CD: Introducing the Setup BoxLang GitHub Action

Mike's Notes

Another excellent reason for migrating to Pipi 10 is to use BoxLang. BoxLang utilises numerous patterns in its code. Pipi excels at generating patterns and is capable of automatically writing GitHub Actions, similar to the examples below.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 
  • Home > Pipi > Pipi 10 > BoxLang

Last Updated

13/06/2025

Streamline Your CI/CD: Introducing the Setup BoxLang GitHub Action

By: Luis Majano
Ortus Solutions: 04/06/2025

Luis Majano is a Computer Engineer and author who has been creating software since the year 2000. He was born in San Salvador, El Salvador in the late 1970s, during a period of economic instability and civil war. He lived in El Salvador until 1995 and then moved to Miami, Florida where he studied and completed his Bachelor of Science in Computer Engineering at Florida International University.

He is the founder and CEO of Ortus Solutions, a consulting firm specializing in web development, ColdFusion (CFML), Java development and all open source professional services under the ColdBox and ContentBox stack. He is the creator of ColdBox, ContentBox, WireBox, MockBox, LogBox and anything BOX, and contributes to many open source ColdFusion/Java projects.

We're excited to announce the release of the Setup BoxLang GitHub Action – a powerful new tool that makes it incredibly easy to integrate BoxLang into your continuous integration and deployment workflows with GitHub actions. Whether you're building applications, running tests, or deploying BoxLang projects, this action eliminates the complexity of environment setup and gets you coding faster.

Why This Matters

Setting up BoxLang in CI environments has traditionally required multiple manual steps: installing Java, downloading BoxLang binaries, configuring paths, and installing necessary modules. With the Setup BoxLang Action, all of this complexity disappears into a single, simple step in your GitHub workflow.

Key Features

  • One-Step Installation: Get BoxLang running in your GitHub Actions workflow with just a few lines of YAML.
  • Automatic Module Management: Install any BoxLang modules you need directly during setup – no additional scripts required.
  • Version Flexibility: Choose from the latest stable release, bleeding-edge snapshots, or pin to specific versions for consistent builds.
  • Zero Configuration: The action automatically handles Java installation and environment setup, so you can focus on your code.

Getting Started

The simplest usage couldn't be easier:

- name: Setup BoxLang
  uses: ortus-boxlang/setup-boxlang@1.0.0

That's it! This single step will install the latest stable version of BoxLang and have it ready for your workflow.

Advanced Usage Examples

Installing Specific Modules

Need AI capabilities, ORM functionality, or PDF generation? Install multiple modules at once:

- name: Setup BoxLang with modules
  uses: ortus-boxlang/setup-boxlang@1.0.0
  with:
    modules: bx-ai bx-orm bx-pdf

Version Control

For production deployments, you might want to pin to a specific version:

- name: Setup BoxLang with specific version
  uses: ortus-boxlang/setup-boxlang@1.0.0
  with:
    version: 1.1.0

Or if you're feeling adventurous and want the latest features:

- name: Setup BoxLang snapshot
  uses: ortus-boxlang/setup-boxlang@1.0.0
  with:
    version: snapshot

Complete Workflow Example

Here's how you might use the Setup BoxLang Action in a real CI workflow:

name: BoxLang CI
on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      
    - name: Setup BoxLang
      uses: ortus-boxlang/setup-boxlang@1.0.0
      with:
        modules: bx-orm bx-pdf
        version: latest
        
    - name: Run tests
      run: boxlang tests.bx
      
    - name: Build application
      run: boxlang Build.bx

System Requirements Made Simple

Don't worry about Java installation – the action automatically installs OpenJDK 21 if it's not already available on the runner. Everything is handled for you behind the scenes.

Available Configuration Options

OpOption Description Default
modules Space-delimited list of BoxLang modules to install None
version BoxLang version (latest, snapshot, or specific version) Latest

Real-World Benefits

  • Faster Onboarding: New team members can contribute immediately without complex local setup procedures.
  • Consistent Environments: Every build runs in the same BoxLang environment, eliminating "works on my machine" issues.
  • Simplified Maintenance: No more maintaining custom installation scripts or Docker images just for BoxLang setup.
  • Module Management: Easily test different module combinations across different branches or environments.

Getting Started Today

The Setup BoxLang Action is available now in the GitHub Marketplace. Simply add it to your workflow file and start building with BoxLang in minutes, not hours.

Visit the ortus-boxlang/setup-boxlang repository for complete documentation, examples, and the latest updates.

Ready to supercharge your BoxLang CI/CD pipeline? Give the Setup BoxLang Action a try and let us know how it improves your development workflow!

Professional Open Source

BoxLang is a professional open-source product, with three different licences:

  • Open-Source Apache2
  • BoxLang +
  • BoxLang ++

BoxLang is free, open-source software under the Apache 2.0 license. We encourage and support community contributions. BoxLang+ and BoxLang ++ are commercial versions offering support and enterprise features. Our licensing model is based on fairness and the golden rule: Do to others as you want them to do to you. No hidden pricing or pricing on cores, RAM, SaaS, multi-domain or ridiculous ways to get your money. Transparent and fair.

BoxLang is more than just a language; it's a movement.

Join us and redefine development on the JVM Ready to learn more? Explore BoxLang's Features, Documentation, and Community.

Join the BoxLang Community

Be part of the movement shaping the future of web development. Stay connected and receive the latest updates on surrounding anything BoxLang

Subscribe to our newsletter for exclusive content.

Follow Us on Social media and don’t miss any news and updates:

What Is Information? The Answer Is a Surprise.

Mike's Notes

A great summary introduction from Quanta Magazine.

Resources

References

  • The Mathematical Theory of Communication (1949). Claude E. Shannon and Warren Weaver.

Repository

  • Home > Ajabbi Research > Library > Subscriptions > Fundamentals from Quanta Magazine
  • Home > Handbook > 

Last Updated

06/06/2025

What Is Information? The Answer Is a Surprise.

By: Ben Brubaker
Fundamentals from Quanta Magazine: 03/06/2025

Ben Brubaker is a staff writer covering computer science for Quanta Magazine. He previously covered physics as a freelance journalist, and his writing has also appeared in Scientific American, Physics Today, and elsewhere. He has a Ph.D. in physics from Yale University and conducted postdoctoral research at the University of Colorado, Boulder..

It’s often said that we live in the information age. But what exactly is information? It seems a more nebulous resource than iron, steam and other key substances that have powered technological transformations. Indeed, information didn’t have a precise meaning until the work of the computer science pioneer Claude Shannon in the 1940s. 

Shannon was inspired by a practical problem: What’s the most efficient way to transmit a message over a communication channel like a telephone line? To answer that question, it’s helpful to reframe it as a game. I choose a random number between 1 and 100, and your goal is to guess the number as quickly as possible by asking yes-or-no questions. “Is the number greater than zero?” is clearly a bad move — you already know that the answer will be yes, so there’s no point in asking. Intuitively, “Is the number greater than 50?” is the best opening move. That’s because the two possible answers are equally likely: Either way, you’ll learn something you couldn’t have predicted. 

In his famous 1948 paper “A Mathematical Theory of Communication,” Shannon devised a formula that translated this intuition into precise mathematical terms, and he showed how the same formula can be used to quantify the information in any message. Roughly speaking, the formula defines information as the number of yes-or-no questions needed to determine the contents of a message. More predictable messages, by this measure, contain less information, while more surprising ones are more informative. Shannon’s information theory laid the mathematical foundation for data storage and transmission methods that are now ubiquitous (including the error correction techniques that I discussed in the August 5, 2024, issue of Fundamentals). It also has more whimsical applications. As Patrick Honner explained in a 2022 column, information theory can help you win at the online word-guessing game Wordle.

In a 2020 essay for Quanta, the electrical engineer David Tse reflected on a curious feature of information theory. Shannon developed his iconic formula to solve a real-world engineering problem, yet the underlying mathematics is so elegant and pervasive that it increasingly seems as if he hit upon something more fundamental. “It’s as if he discovered the universe’s laws of communication, rather than inventing them,” Tse wrote. Indeed, Shannon’s information theory has turned out to have unexpected connections to many different subjects in physics and biology.

What’s New and Noteworthy

The first surprising link between information theory and physics was already present in Shannon’s seminal paper. Shannon had previously discussed his theory with the legendary mathematician John von Neumann, who observed that Shannon’s formula for information resembled the formula for a mysterious quantity called entropy that plays a central role in the laws of thermodynamics. Last year, Zack Savitsky traced the history of entropy from its origins in the physics of steam engines to the nanoscale “information engines” that researchers are developing today. It’s a beautiful piece of science writing that also explores the philosophical implications of introducing information — an inherently subjective quantity — into the laws of physics.

Such philosophical questions are especially relevant for researchers studying quantum theory. The laws of quantum physics were devised in the 1920s to explain the behavior of atoms and molecules. But in the past few decades, researchers have realized that it’s possible to derive all the same laws from principles that don’t seem to have anything to do with physics — instead, they’re based on information. In 2017, Philip Ball explored what researchers have learned from these attempts to rebuild quantum theory.

Physics isn’t the only field influenced by ideas from information theory. Soon after Shannon’s paper, information became central to the way researchers think about genetics. More recently, some researchers have brought principles from information theory to bear on some of the thorniest questions in biology. In a 2015 Q&A with Kevin Hartnett, the biologist Christoph Adami described how he uses information theory to explore the origins of life. In April, Ball wrote about a new effort to reframe biological evolution as a special case of a more fundamental “functional information theory” that drives the emergence of complexity in the universe. This theory is still speculative, but it illustrates the striking extent of information theory’s influence.

As the astrobiologist Michael Wong told Ball, “Information itself might be a vital parameter of the cosmos, similar to mass, charge and energy.” One thing seems certain: Researchers studying information can surely expect more surprises in the coming years.

Unix Mindset: MCP Is Unix Pipes for AI

Mike's Notes

"The Model Context Protocol (MCP) is an open standard, open-source framework introduced by Anthropic to standardise the way artificial intelligence (AI) models like large language models (LLMs) integrate and share data with external tools, systems, and data sources. Technology writers have dubbed MCP “the USB-C of AI apps”, underscoring its goal of serving as a universal connector between language-model agents and external software. Designed to standardise context exchange between AI assistants and software environments, MCP provides a model-agnostic universal interface for reading files, executing functions, and handling contextual prompts. It was officially announced and open-sourced by Anthropic in November 2024, with subsequent adoption by major AI providers including OpenAI and Google DeepMind." - Wikipedia

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

28/05/2026

Unix Mindset: MCP Is Unix Pipes for AI

By: Kingsley Uyi Idehen
LinkedIn: 01/06/2025

Founder & CEO at OpenLink Software | Driving GenAI-Based AI Agents | Harmonizing Disparate Data Spaces (Databases, Knowledge Bases/Graphs, and File System Documents).

The “Unix mindset applied to AI” is a compelling paradigm that draws from Unix’s foundational design philosophy and applies it to AI systems architecture. It’s a powerful insight I came across while digesting a recent presentation by Reuven Cohen.

Here’s how Unix pipes and the Model Context Protocol (MCP) embody this approach:

Unix Philosophy Core Tenets

The Unix philosophy revolves around several key principles:

  1. Do one thing well – Build small, focused tools instead of monolithic applications
  2. Composability – Chain simple tools to create complex workflows
  3. Universal interface – Use text streams as a common data format
  4. Modularity – Favor loosely coupled components that can be mixed and matched

Unix Pipes as the Model

Unix pipes (|) are the quintessential example of this philosophy. You can write:

cat data.txt | grep "error" | sort | uniq -c | head -10

Each tool (cat, grep, sort, uniq, head) performs one task exceptionally well. Together, they form a powerful, composable data-processing pipeline. The magic lies in composition, not in any single tool.

MCP Protocol: Pipes for AI

The Model Context Protocol extends this Unix mindset into the world of AI:

  1. Standardized Interfaces – Like Unix pipes use text, MCP uses standardized JSON-RPC protocols for AI-tool communication—providing a universal interface for AI systems to interact with services and data.
  2. Composable AI Workflows – Rather than building monolithic AI systems, you can compose:Data connectors (to databases, APIs, file systems)Processing tools (calculators, web scrapers, code interpreters)Specialized models (e.g., for vision, reasoning, code generation)Output formatters (to generate documents, charts, or dashboards)
  3. Tool Interoperability – Vendors can create MCP-compatible tools that work together seamlessly—just like Unix tools from different sources pipe into one another without friction.

Practical Applications

This enables streamlined, reusable AI workflows such as:

Document → Semantic Analysis → Content Transformation → Data Lookup → Formatting → Publication 

Each step is a focused component that adheres to MCP protocols. You’re not locked into a single vendor’s ecosystem—you’re free to mix the best tools for the job.

The Broader Vision

This marks a shift from AI as a black box to AI as composable infrastructure—where intelligence is modular, interoperable, and infinitely reusable, just like the Unix tools that have underpinned computing for decades.

BTW — Google Gemini’s Canvas now includes an HTML-based infographic generator, which I used to create an interactive visual version of this concept. It also includes rich metadata, offering yet another showcase of the powerful symbiosis between recent Large Language Model (LLM) innovations and the long-established (and now increasingly appreciated) power of structured data representation—rooted in the same conceptual tenets that gave rise to the World Wide Web: Linked Data Principles (where entities and entity relationship types a named using hyperlinks).

View of the Infographic version of this article using the OpenLink Data Sniffer Browser extension for discovering and visualizing document metadata

You can view the Infographic by clicking on the link below:

  • Unix Mindset, Pipes & MCP for AI: An Infographic

This is the kind of flexibility and power our Virtuoso platform delivers—seamlessly combining Data Spaces with a full-featured Web Application Server.

If you haven’t yet explored Virtuoso—or its new OpenLink AI Layer (OPAL) add-on—you’re missing a direct path to harnessing the transformative potential of AI that’s redefining the future of software.

Why wait? The future is composable, interoperable, and agentic—and Virtuoso gets you there, faster.

How Math is Visual - by Scientific American

Mike's Notes

A great demonstration of visual thinking as a discovery tool in maths.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library > Authors > Benoit Mandelbrot
  • Home > Handbook > 

Last Updated

04/06/2025

How Math is Visual - by Scientific American

By: Marissa Fessenden
Scientific American: 03/01/2013

Papers from Benoit Mandelbrot's office offer a peek into the mathematician's thinking process. His work and that of his contemporaries show how images can inform theory and discovery.

Markov Chain Monte Carlo: Made Simple Once and For All

Mike's Notes

A bit of helpful background.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library > Subscriptions > Towards Data Science
  • Home > Handbook > 

Last Updated

24/01/2026

Markov Chain Monte Carlo: Made Simple Once and For All

By: Pol Marin
Towards Data Science: 01/03/2024

Introduction to MCMC, dividing it into its simplest terms.

I recently posted an article where I used Bayesian Inference and Markov chain Monte Carlo (MCMC) to predict the CL round of 16 winners. There, I tried to explain Bayesian Statistics in relative depth but I didn’t tell much about MCMC to avoid making it excessively large. The post:

  • Using Bayesian Modelling to Predict The Champions League

So I decided to dedicate a full post to introduce Markov Chain Monte Carlo methods for anyone interested in learning how they work mathematically and when they prove to be useful.

To tackle this post, I’ll adopt the divide-and-conquer strategy: divide the term into its simplest terms and explain them individually to then solve the big picture. So this is what we’ll go through:

  • Monte Carlo methods
  • Stochastic processes
  • Markov Chain
  • MCMC

Monte Carlo Methods

A Monte Carlo method or simulation is a type of computational algorithm that consists of using sampling numbers repeatedly to obtain numerical results in the form of the likelihood of a range of results of occurring.

In other words, a Monte Carlo simulation is used to estimate or approximate the possible outcomes or distribution of an uncertain event.

A simple example to illustrate this is by rolling two dice and adding their values. We could easily compute the probability of each outcome but we could also use Monte Carlo methods to simulate 5,000 dice-rollings (or more) and get the underlying distribution.

Stochastic Processes

Wikipedia’s definition is "A stochastic or random process can be defined as a collection of random variables that is indexed by some mathematical set"[1].

In more readable terms: "it’s any mathematical process that can be modeled with a family of random variables".[2]

Let’s use a simple example to understand the concept. Imagine you put a video camera on your favorite store to, once every 2 minutes, check how many visitors there are. We define X(0) as the initial state and it shows the number of visitors seen at t=0. Then, 2 minutes later, we see X(1) and so on.

The state space is a set of values that our random variables (X(i)) can adopt, and they can get from 1 to the maximum store capacity.

One of the properties of a stochastic process is that whatever happens in a specific moment is conditioned to what has happened in the preceding moments. Keeping up with our example, if we have 100 visitors at t=0, the probability of having 100 ± 20 at t=1 is greater than seeing it drop to 10, for example (if no unexpected event happens). Therefore, these X variables aren’t independent.

Markov Chains

A Markov chain is a sequence of numbers where each number is dependent on the previous value of the sequence.

So it’s a stochastic method with one peculiarity: knowing the current state is as good as knowing the entire history. In mathematical terms, we say that a stochastic process is Markovian if X(t+1) conditioned to x(1), x(2),…x(t) only depends on x(t):

Mathematical expression – Image by the author

Keeping up with our example, for it to be considered a Markov chain we would need the number of visitors in a given time – t – to only depend on the number of visitors we saw in the previous instant – t-1. That’s not true in real life but imagine it is, then we define the transition probability as the probability of going from state i to state j in a specific instant:

Transition probability in a Markov chain – Image by the author

And, if that probability is time-independent, we say it’s stationary.

With this transition probability, we now define the transition matrix, which is just a matrix with all transition probabilities:

Markovian transition matrix – Image by the author

This matrix comes in handy when we want to compute the probabilities of transitioning from one state to another in n steps, which is achieved mathematically with power operations on the matrix:


Power operation on the matrix to get transition probability after n steps – Image by the author

Let’s define now a new – and dumb – example, in which we consider that a striker’s probability of scoring a goal in a football (soccer) match depends only on whether he/she scored in the previous game or not. Because we suppose it’s also time-independent – when the match is played doesn’t matter – we are working with stationary transition probabilities.

Concretely, if a player scored in the previous match, we assume the probability of scoring again in the next game is 70% (the player is hypermotivated to keep the streak going). If the player doesn’t score, this probability drops to 40%.

Let’s put that into the transition matrix:

Transition matrix for our example – Image by the author

The proper way to read it is: we have two possible outcomes (goal or no goal). Row 1 defines the next game probabilities for the case in which the player has scored; row 2 does the same but for the case in which he/she hasn’t scored. Columns are read similarly: the first one relates to the probabilities of scoring and the second to the probabilities of not scoring.

So, for example, 0.7 is the probability of scoring after having scored in the previous game.

Now, what are the chances that a certain player scores in the n = 2 game knowing that he hasn’t scored today?

Transition matrix for n=2 – Image by the author

If the player hasn’t scored today, we have to focus on the second row. As we’re interested in the chances of scoring, we focus on the first column. And where these both intersect we have 0.52–the probability of scoring in the game ahead of the next one is 52%.

We could want to work out the marginal distributions for each X(t) and we can do it by using the initial conditions in which the chain initialized: X(0).

Keeping up with the example, the question would now be: knowing that the player has a 50–50% chance of scoring in the first game, what are the chances that he/she scores then and in the second game ahead of the first?

Transition matrix with marginal distributions – Image by the author

The answer is 0.565, or 56.5%.

What’s curious about Markov Chains is that, independently of which values we choose for p0, we might end up with the same distribution after a certain number of iterations. That’s called a stationary distribution, and this is key for MCMC.

Markov Chain Monte Carlo (MCMC)

Now it’s time to combine both methods together.

MCMC methods constitute Monte Carlo simulations where the samples are drawn from random Markov chain sequences to form a probability distribution. In the case of Bayesian modelling, this stationary distribution will be the posterior distribution.

Simulating the chain after a given set of steps (what’s called the burn-in phase) we’ll get us to the desired distribution. These simulations are dependent on each other but, if we discard a few after certain iterations, we make sure these simulations are almost independent (thinning).

MCMC comes in handy when we want to perform inference for probability distributions where independent samples from the distribution cannot be easily drawn.

Regarding the different MCMC algorithms that exist, we’ll focus on the two more common ones:

Gibbs Sampling: this algorithm for sampling samples from the conditional distributions. Here, we sample our variables based on the distribution conditional to the other variables and iteratively repeat this process. For example, in a case where we have 3 variables, we would simulate the first one by sampling, for each t in 1…N iterations:


Variable update using conditional distribution – Image by the author

Metropolis-Hastings: is usually the alternative to Gibbs when simulating the complete conditionals isn’t possible (i.e. when we cannot sample a variable conditioned to all the other ones). This works by proposing a candidate for the next step in the Markov chain – x(cand) – by sampling from a simple distribution – q – built around x(t-1). Then we choose to accept the candidate or not with a determined probability (if it’s not accepted, then the chain doesn’t change). This probability is defined by:

Acceptance probability in Metropolis-Hastings – Image by the author

Conclusion

In short, MCMC methods consist of drawing random samples conditioned to the previous value/step only and potentially deciding whether we keep them or not. And repeat multiple times until we form the chains.

  1. To schematize, let’s define the algorithm in a set of steps:
  2. Get/Assign the initial values.

For each iteration: a) Sample the candidates from a distribution that only depends on the previous value (Markov Chain). b) If we’re using the Metropolis-Hastings algorithm, decide whether we accept or reject the candidates by computing and using the acceptance probability. c) Update/Store the new values.

Resources

[1] Wikipedia contributors. (2024, January 8). Stochastic process. In Wikipedia, The Free Encyclopedia. Retrieved 19:43, February 24, 2024, from https://en.wikipedia.org/w/index.php?title=Stochastic_process&oldid=1194369849

[2] Christopher Kazakis. (2021, January 8th). See the Future with Stochastic Processes. Retrieved 19:43, February 24, 2024, from https://towardsdatascience.com/stochastic-processes-a-beginners-guide-3f42fa9941b5.