Data Contracts

Mike's Notes

I need to think about data contracts. I have an API Engine (API) to sort out soon, and many data exchanges are also coming up. What should I do?

Data contracts bring data providers and data consumers together.

A data contract is a document that defines the structure, format, semantics, quality, and terms of use for exchanging data between a data provider and its consumers. Think of an API, but for data. A data contract is implemented by a data product or other data technologies, even legacy data warehouses. Data contracts can also be used for the input port to specify the expectations of data dependencies and verify given guarantees.

Below is an article I found on Data Mesh Manager.

Resources

References


Repository

  • Home > Ajabbi Research > Library 

Last Updated

28/03/2025

What is a Data Contract?

Data Mesh Manager:

A data contract defines the structure, format, semantics, quality, and terms of use for exchanging data between a data provider and their consumers. A data contract is implemented by a data product’s output port or other data technologies. Data contracts can also be used for the input port to specify the expectations of data dependencies and verify given guarantees.

Data provider on the left, data consumer on the right, data contract in the middle

Data Contract Example

Let's start with an example to see, what a data contract typically covers:


Screenshot of a data contract in Data Mesh Manager

Before we dive into all the details, let's discuss, how we use data contracts.

Collaboration

Data contracts come into play when data is exchanged between different teams or organizational units, such as in a data mesh architecture. First, and foremost, data contracts are a communication tool to express a common understanding of how data should be structured and interpreted. They make semantic and quality expectations explicit. They are often created collaboratively in workshops together with data providers and data consumers, even before the data product is implemented. This is what we call contract-first. Later in development and production, they also serve as the basis for code generation, testing, schema validations, quality checks, monitoring, access control, and computational governance policies.

In a data contract, all attributes of a data model are precisely described and defined with their syntax and semantics. This can be done in the form of technology-neutral way or a technology-specific schema (e.g., SQL DDL, dbt model, Protobuf, JSON Schema), or both. These serve as fixed points for the provider team, which must always be adhered to. But it is also gives flexibility for the providing team, how to design and implement internal components of a data product to fulfill this interface. Consumers can trust that the fields are stable and meet the defined quality standards.

Approval Process

Note

The term data contract does not align with a contract in a legal sense as a mutual agreement between two parties. A data contract specifies the provided data set and is owned by one party, typically the data provider. So, the term data contract may be somewhat misleading, but it is how it is used in practice. The mutual agreement between one data provider and one data consumer is the data usage agreement that refers to a data contract.

The bilateral agreement is reached through an approval process:

A consumer team interested in using the data submits a request to access the data product of another team (provider team). It states the purpose of the intended data usage. The provider team then decides whether to approve the request, based on criteria such as the usage terms, a valid purpose, and need-to-know principle, or whether to need to negotiate anything further with the consumer team. When the request is approved, a data usage agreement is concluded with the data consumer.

A data usage agreement has a life-cycle with a start date, and it can be canceled by either party with respect to a defined notice period. This makes it possible for the provider team to evolve a data product, e.g., when a breaking change needs to be implemented and data consumers are advised to migrate to newer version of an output port within the notice period. Data consumers can also cancel a data product, e.g., when the costs don't meet the expected business value or the data quality is not sufficient. This embraces product-thinking: Data providers need to make sure that their consumers gain value by stable and high-quality data.

Automation and Contract Enforcement

The request flow and lifecycle processes should be fully implemented as a self-service, in line with the data mesh principles. This also includes processes and notifications for new requests, approvals, reassessments, and terminations.

Data contracts and data usage agreements can further act as the foundation to automate processes in the data platform and for computational governance: As soon as a data usage agreement has been approved and within the start date, permissions for the respective data product can be set up automatically in the data platform. When the agreement is terminated, the permissions are revoked.

Data contracts can also be the basis to perform automated tests and quality checks in the CI/CD pipeline or in data quality monitoring tools. For example, a regular check ensures that the schema conforms to the agreed data model and the data meets the defined quality attributes.

Data Contract Specification

For automation, a data contract must be available in a machine-readable form, such as a YAML representation. This is why we propose the Data Contract Specification:

[IMG]

The example from above, encoded as YAML:

dataContractSpecification: 0.9.1
id: urn:datacontract:checkout:snowflake_orders_npii_v2
info:
  title: snowflake_orders_npii_v2
  version: 1.0.0
  description: "All order-created events, PII removed."
  owner: checkout
  contact: {}
terms:
  usage: Max. 10x queries per day
  limitations: Not suitable for real-time use cases
  billing: $1000 / month
  noticePeriod: P3M
models:
  orders:
    type: table
    description: Table containing order information with masked PII.
    fields:
      order_id:
        type: text
        description: Unique identifier for the order.
      customer_id:
        type: text
        description: Unique identifier for the customer.
      email:
        type: text
        description: Masked email address of the customer.
      phone_number:
        type: text
        description: Masked phone number of the customer.
      order_date:
        type: timestamp
        description: Date of the order.
      order_total:
        type: decimal
        description: Total amount of the order.
schema:
  type: sql-ddl
  specification: |-
    CREATE TABLE orders (
        order_id STRING COMMENT 'Unique identifier for the order.',
        customer_id STRING COMMENT 'Unique identifier for the customer.',
        email STRING COMMENT 'Masked email address of the customer.',
        phone_number STRING COMMENT 'Masked phone number of the customer.',
number of the customer.',
        order_date TIMESTAMP_TZ COMMENT 'Date of the order.',
        oder_total DECIMAL COMMENT 'Total amount of the order.'
    );
examples:
- type: csv
  description: Randomly generated values via ChatGPT
  data: |-
order_id,customer_id,email,phone_number,order_date,order_total
    1,101,masked_email_1,masked_phone_1,2023-07-01,100.50
    2,102,masked_email_2,masked_phone_2,2023-07-02,75.25
    3,103,masked_email_3,masked_phone_3,2023-07-03,50.00
    4,104,masked_email_4,masked_phone_4,2023-07-04,200.20
    5,105,masked_email_5,masked_phone_5,2023-07-05,300.75
    6,106,masked_email_6,masked_phone_6,2023-07-06,120.80
    7,107,masked_email_7,masked_phone_7,2023-07-07,50.50
    8,108,masked_email_8,masked_phone_8,2023-07-08,90.00
    9,109,masked_email_9,masked_phone_9,2023-07-09,180.6
10,110,masked_email_10,masked_phone_10,2023-07-10,250.40
quality:
  type: custom
  specification: |-
    quality_checks:
      - name: Check for Null Values
        description: Ensure that there are no missing values in critical columns.
        sql: |
          SELECT *
          FROM orders
          WHERE order_id IS NULL
             OR customer_id IS NULL
             OR email IS NULL
             OR phone_number IS NULL
             OR order_date IS NULL
             OR order_total IS NULL;
      - name: Check for Duplicates
        description: Ensure that each order has a unique identifier, and there are no duplicate entries in the "orders" table.
        sql: |
          SELECT order_id
          FROM orders
          GROUP BY order_id
          HAVING COUNT(*) > 1
      - name: Check for Valid Email Addresses
        description: Ensure that the "email" column contains valid email addresses using a standard email pattern.
        sql: |
          SELECT *
          FROM orders
          WHERE NOT REGEXP_LIKE(email, '[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}');
      - name: Check for Positive Order Total
        description: Ensure that the "order_total" column contains positive values, indicating valid order amounts.
        sql: |
          SELECT *
          FROM orders
          WHERE order_total < 0;
      - name: Check Order Date Within a Reasonable Range
        description: Ensure that the "order_date" falls within the range from January 1, 2020, to December 31, 2023, to verify the dates are within a reasonable timeframe.
        sql: |
          SELECT *
          FROM orders
          WHERE order_date < '2020-01-01' OR order_date > '2023-12-31';

The example follows the Data Contract Specification that is compatible with Data Mesh Manager's Data Contract API.

The YAML can be read by tools, such as the Data Contract CLI to automate code generation, detect breaking changes in the CI/CD pipeline and to trigger other tools, such as Soda Core Engine to validate quality attributes on the actual data sets.

Visualize the Data Mesh

Data contracts and data usage agreements are also powerful for data discovery and data lineage. You can think of a data mesh architecture as a graph: Data products are the nodes, and data usage agreements represent the edges between data products. With that, the data mesh can be visualized as a map:


The mesh visualized as a map

Such a data map is a way of making the use of data in the company comprehensible and traceable across teams and domains.

Data Mesh Manager

Data contracts need to be managed efficiently and comprehensibly. Many of our customers used a wiki for this purpose, but this quickly reaches its limits and enables hardly any automation.

Since there was no other good tool available for managing data contracts and data usage agreements, we developed Data Mesh Manager, to manage data products, data contracts, and global policies as a web-based self-service. An event-based API enables seamless integration with any data platform. And any change will be recorded in an audit trail.


Screenshot of a data contract in Data Mesh Manager

In addition to a data product inventory for finding and evaluating data products, the Data Mesh Manager also supports a request and accept flow for creating data usage agreements, as well as an event-based API for automatically creating and revoking permissions in the data platform. The visualization as a data map makes the mesh comprehensive and the use of the data products traceable.

Building High-Performing Innovation Teams: Commitment vs. Consensus

Mike's Notes

Jeremiah Gardner on commitment vs consensus. I have seen the problem he is describing many times.

Resources

References


Repository

  • Home > Handbook > Teams

Last Updated

26/03/2025

Building High-Performing Innovation Teams: Commitment vs. Consensus

By: Jeremiah Gardner
jeremiahgardner.com: 26/03/2025

Let's be honest about something – working on an innovation team is hard. Really hard.

Anyone telling you differently is either selling something or hasn't actually done it.

Here's why: Not only are you trying to find answers to questions nobody's asked before (that's challenge number one), but you're doing it with a group of humans who each bring their own perspectives, biases, and ways of working. That's your one-two punch right there.

And what do most teams do when faced with this complexity? They chase consensus like it's the holy grail of innovation.

You know how it goes. The endless meetings. The careful tiptoeing around disagreements. The diplomatic dance of making sure everyone feels heard. The desperate pursuit of harmony at all costs.

Sounds nice, right? Almost utopian.

But here's the thing – while we're all sitting in our comfy meeting rooms, nodding along and "building consensus," something critical is happening: absolutely nothing.

Let me paint you a picture of what consensus actually looks like in practice:

Three-hour meetings that could've been 30-minute decisions

"Let's circle back" becoming your team's unofficial motto

Death by a thousand tiny compromises

That gnawing feeling that you're moving at the speed of bureaucracy

Meanwhile, your customers – you know, the people you're supposedly innovating for – are out there, waiting for solutions while you're debating the finer points of your project timeline in meeting room B.

This is where great innovation teams zag while others zig. They understand a fundamental truth:

Consensus is the enemy of progress. Commitment is the ally of innovation.

What do I mean by commitment? It's not about blind agreement or hierarchical mandate. It's about:

  • Crystal clear priorities that everyone understands
  • Quick, informed decisions that keep momentum alive
  • Forward movement without the paralysis of second-guessing
  • Actual accountability (yes, with real names attached to real tasks)

I've seen this play out countless times in my work with Fortune 500 innovation teams. The most successful ones aren't the ones with perfect harmony – they're the ones who've mastered the art of commitment.

Take Netflix's approach to innovation teams. They famously operate on what they call "informed captains" – team leaders who are expected to make clear decisions after gathering input, rather than seeking consensus. As former Netflix executive Patty McCord puts it, "Good teams don't wait for consensus. They listen, debate, and then commit to a course of action."

Want to shift your team from consensus-seeking to commitment-driving? Start with these four questions in your next meeting:

  • "What specifically do we need to learn next?"
  • "How exactly will we learn it?"
  • "Who's taking point on this?" (Yes, actual names)
  • "When do we regroup to share learnings?"
  • Then – and this is crucial – get out of the building and go learn something from your actual customers.

Remember: Innovation isn't a committee sport. It's a commitment game. The teams that win aren't the ones who agree on everything – they're the ones who commit to learning fast and moving forward together.

Ready to ditch consensus and embrace commitment? Your customers are waiting.

HTTP 2 vs HTTP 3 — What's the Difference?

Mike's Notes

Useful background information.

Resources

References


Repository

  • Home > Ajabbi Research > Library > Subscriptions > Level Up Coding

Last Updated

23/03/2025

HTTP 2 vs HTTP 3 — What's the Difference?

By: Nikky Siapno
Level Up Coding: 23/03/2025

HTTP 1 started in 1996. The very next year HTTP 1.1 followed. It was another ~20 years until HTTP 2 became standardized in 2015. And in recent years (2022), HTTP 3 was officially standardized.

But what’s the difference?

Starting at the foundation:

HTTP 1.1:

  • Persistent connections — Reuses connections instead of opening new ones
  • Chunked transfers — Sends data in parts instead of waiting for the full response
  • Improved caching — Introduced headers for better caching and connection management
  • Sequential requests — Requests block each other (HoL blocking at the request level)
  • Multiple connections needed — Browsers used multiple TCP connections for speed
  • It introduced core features still used today.

HTTP 2:

  • Multiplexing — Multiple requests in a single TCP connection
  • Header compression (HPACK) — Reduces metadata size
  • Stream prioritization — Ensures critical resources load first
  • Head-of-line (HoL) blocking — A lost packet blocks all streams
  • While HTTP 2 optimized TCP, it remained constrained by TCP’s head-of-line blocking.

HTTP 3:

  • Built on QUIC (UDP) — No more TCP bottlenecks
  • Independent streams — Packet loss in one stream doesn’t affect others
  • Faster handshakes — Combines transport + encryption setup in one step
  • Mandatory encryption (TLS 1.3) — Security by default
  • Connection migration — Seamless across network changes

In a nutshell: HTTP 2 optimized TCP, but HTTP 3 rewrites the game with QUIC, making it faster, more reliable, and encrypted by default.

Which fact surprised you?

How Does SQL Execution Order Work, and Why is it so Important?

Mike's Notes

Level Up Coding is a great visual reference.

Resources

References


Repository

  • Home > Ajabbi Research > Library > Subscriptions > Level Up Coding

Last Updated

22/03/2025

How Does SQL Execution Order Work, and Why is it so Important? 

By: Nikki Siapno
Level Up Coding: 02/03/2025

A SQL query executes its statements in the following order:

  • FROM / JOIN
  • WHERE
  • GROUP BY
  • HAVING
  • SELECT
  • DISTINCT
  • ORDER BY
  • LIMIT / OFFSET

The techniques you implement at each step help speed up the following steps. This is why it's important to know their execution order. To maximize efficiency, focus on optimizing the steps earlier in the query.

With that in mind, let's take a look at some optimization tips:

1) Maximise the WHERE clause

This clause I executed early, so it's a good opportunity to reduce the size of your data set before the rest of the query is processed.

2) Filter your rows before a JOIN

Although the FROM/JOIN occurs first, you can still limit the rows. To limit the number of rows you are joining, use a subquery in the FROM statement instead of a table.

3) Use WHERE over HAVING

The HAVING clause is executed after WHERE & GROUP BY. This means you're better off moving any appropriate conditions to the WHERE clause when you can.

4) Don't confuse LIMIT, OFFSET, and DISTINCT for optimization techniques

It's easy to assume that these would boost performance by minimizing the data set, but this isn’t the case. Because they occur at the end of the query, they make little to no impact on its performance.

If you want to create efficient queries, it's a good idea to understand how things work under the hood otherwise your efforts may be wasted. While these tips work best in most cases, you should consider your unique use case when choosing the best course of action.

I broke the CMS

Mike's Notes

The best way to learn is to make mistakes. Oops.

Resources

References


Repository

  • Home > PipiWiki > Engines

Last Updated

24/03/2025

I broke the CMS

By: Mike Peters
On a Sandy Beach: 21/03/2025

Mike is the inventor and architect of Pipi and the founder of Ajabbi.

The integration experiment running live data through the Plug-in Engine (plu) and Learning Object Engine (lob) with the Content Management Engine (cms) didn't work.

It led me to discover a previously hidden issue with the CMS data model, which yesterday led to the model's change and the CMS's now-broken state.

To get the CMS working again, a lot of code must be revised. The Template Engine (tem) and the Render Engine (rnd) will also need minor altering.

However, with these complex changes, the integration to enable plug-ins and learning objects should now work in future.

I have been using a temporary workaround to successfully work with all 3 engines.

Plug-ins are necessary for two current projects;

  • my first customer needs live-sign language interpreters via Relay (a video service) embedded on their static website
  • Embedding Wolfram Notebooks on Pipi documentation pages
Learning objects are necessary for these tasks on the roadmap;
  • User documentation
  • the UI help system
I also realised that adding plug-ins and learning objects is not on the published roadmap. A job that is now done. (Roadmap wbs 4.1 - 4.5)

Nikolai Vavilov and the Living Library of Resilience: The Story of the World’s First Seed Bank and the Tragic Hero of Science Who Set Out to End Humanity’s Suffering

Mike's Notes

Maria Popova tells a moving story about Nikolai Vavilov, one of humanity's heroes. If you understand his motivations, you will understand the why behind Ajabbi.

Resources

References


Repository

  • Home > Ajabbi Research  > Library > Subscriptions > The Marginalian

Last Updated

20/03/2025

Nikolai Vavilov and the Living Library of Resilience: The Story of the World’s First Seed Bank and the Tragic Hero of Science Who Set Out to End Humanity’s Suffering

By: Maria Popova
The Marginalian: 8/03/2023

I spent large swaths of my childhood by my grandmother’s side in rural Bulgaria as she tended to her subsistence garden, tilling and planting, watering and weeding. Each August, we did something that felt to me like partaking of magic — we would choose the sweetest, most succulent tomatoes from the vine, cut them open, carefully extract the seeds, and lay them out on newspaper to dry, knowing that they would become next spring’s seedlings and, with nothing more than sunlight and water, next summer’s bright red orbs of delight. So it is that, year after year, my grandmother refined her tomatoes into a cornucopia of unparalleled sweetness and perfection. Last summer’s seeds are already growing as I write.

This magic was made possible by a visionary of science who set out to save humanity and died for his values the year my grandmother turned nine.


Tomato, or Love-Apple, from Elizabeth Blackwell’s pioneering 1737 encyclopedia of medicinal plants. (Available as a print, benefitting The Nature Conservancy.)

While the physicist Sergei Vavilov was presiding over Stalin’s Academy of Sciences and spearheading the Soviet atomic bomb project, his idealistic older brother was laboring at something of orthogonal impact on humanity — a way to end an elemental form of suffering that has haunted our species since its dawn.

The botanist, geneticist, and explorer Nikolai Vavilov (November 25, 1887–January 26, 1943) was still a boy when he arrived at his dream of ending famine. He had heard his father’s stories of growing up in poverty and constant hunger due to crop failures. When Nikolai himself was four, the early arrival of winter decimated crops all over the country, sending millions into starvation. All the tsar could do was offer his subjects “famine bread” — loaves made of milled husks, bark, weeds, and moss, rationed out in the freezing cold. Vavilov’s father had spent his life rising from poverty and now had a comfortable life as a merchant, so the family was protected from the worst of the famine — but from his precarious island of comfort, the boy watched the ocean of suffering and sorrowed. Half a million peasants perished that winter as the aristocracy feasted on imported delicacies from Europe — grim structural inequality that became the ignition spark for the long-seething people’s revolution a quarter century later.

Vavilov saw the contours of a different kind of revolution — one no one else could envision, not in Russia and not anywhere in the world.


Nikolai Vavilov

He wrote in the diary of his youth:

Do what you can. If you can’t do something you wanted to do, then you will be forgiven, but if you don’t want to try to do anything, you will not be forgiven.

He decided to do nothing less than end the world’s hunger, vowing in his diary to devote his life to science — an endeavor aimed at “everything that brings joy, calmness of emotion and reason” — so that he may “understanding nature for the betterment of humankind.”

After graduating from the Soviet agricultural academy as a botanist, he set out to travel through Europe and absorb all he could from the best scientists in every related discipline. In England, he worked with William Bateson, who had coined the word genetics to explain heredity and had pioneered the study of this script for transmitting the message of life.

Upon returning to Russia, Vavilov founded an institute under which to commence the great project of his life — collaborating with nature on enhancing her strengths and allaying her weaknesses by using the new science of genetics to cultivate plant species that would thrive in conditions none had survived before. He had a revolutionary insight: There must be wild varieties of common agricultural plants with different genes that make them more resilient than their farmed cousins — genes that could be used to strengthen agricultural crops by breeding stronger species that would feed humanity even through droughts and freezes. He called them his miracle plants. It wasn’t just an idealist’s dream — he knew the science that would make it a reality, and he would devote his life to it.

When World War I broke out, Vavilov, already established as a preeminent botanist, was dispatched to present-day Iran to solve a mystery — Soviet soldiers there were suffering from brain fog and inexplicable dizziness. He discovered that the mysterious malady was caused by a fungus growing on the wheat of which their bread was made. As bullets flew around him, Vavilov carefully collected samples of local plants, wrapped them in wax paper, and tucked them into his breast pocket. He didn’t yet know it, but this was the birth of Earth’s largest botanical collection.


The pea by French artist Paul Sougy. (Available as a print, benefitting The Nature Conservancy.)

When a drought lashed Russia in 1921 and killed the harvest, more than 5 million people died of starvation in a year, most of them peasants. Vavilov grew determined to never let this happen to anyone again. He understood that if he could equip farmers with the basic science of genetics, they could control for which traits of their crops would dominate, rather than entrusting their harvest to the roulette of chance — they could do what my grandmother did with her tomatoes, selecting for the best traits year over year. Mendel had made a science of agriculture by expressing mathematically the probabilities of genetic variance. Vavilov set out to make of that science an art of resilience, having vowed as a young man to “work for the benefit of the poor, the enslaved class of my country, to raise their level of knowledge.”

He spent the 1920s roaming the world to collect wild varieties of staple foods. He slept little, smiled much, and trekked through the jungle in his tailored three-piece suit, tie, and felt fedora. He traveled to places frequented by droughts and food shortages, from Africa to the Middle East, taking care to learn the language and talk to locals about their lore of growing food in inhospitable conditions. He traveled to the birthplaces of the most nutritious plants. In Brazil, he got cacao, oranges, mangoes, and papayas. In China, poppy and sugarcane. In Korea, soybeans and rice. In Ethiopia, he discovered the mother plant from which all the world’s coffee originated.


Cacao by Étienne Denisse from his Flore d’Amérique, 1846. (Available as a print, a cutting board, and stationery cards, benefitting The Nature Conservancy.)

By the end of the decade, Vavilov had completed numerous ethnobotanical expeditions to collect hundreds of thousands of seeds from five continents, including many places where no scientist had set foot before. He was quietly building something unexampled: the world’s first seed bank — a living library of biodiversity that would come to the rescue of the people of any land whose crops were decimated by a drought or a blight. There were 600 kinds of apples and more than a thousand varieties of strawberries among its quarter million plants — a lush repository of resilience, housed at Vavilov’s institute in Leningrad.

Lenin, who had assumed power in the 1917 Russian Revolution, had immediately recognized the political value of Vavilov’s humanistic work — its insurance against the country’s crop failures, its promise of making Russia a superpower of global food production — and had thrown his full support behind it. But when he died in 1924, everything changed.

As Stalin usurped power, he forced peasant farmers off their farms and into large industrial agriculture collectives — tumult that disrupted the harvest and hurled the country into mass starvation. He knew that a widespread famine would hamper his revolution; he knew that more resilient crops would be the solution. But it was not Vavilov’s science he turned to.

On August 7, 1927, Pravda — the newspaper voice of the Communist Party — published a fawning profile of a young “barefoot scientist” in rural Azerbaijan who had never gone to university but was promising an agrarian revolution.

Trofim Lysenko considered scientific education “harmful nonsense.” He rejected Darwinian evolution and Mendelian genetics, instead subscribing to Lamarckian inheritance with its outlandish claim that organisms acquire traits in immediate response to their environments and pass those traits immediately to the next generation — a pseudoscience that fueled the menace of eugenics. There were echoes of alchemy in Lysenko’s bravado — he promised he could cultivate wheat that would turn into rye and rye that would turn into barley. He bragged that his pea crop had withstood winter thanks to an innovative “training” strategy — soaking the seeds in ice-cold water, which he called vernalization. He claimed he could “train” plants within a single generation, making the very next generation more resilient.


Trofim Lysenko measuring wheat

Stalin, having no understanding of science, was blinded by the luster of the young man’s instant gratification claims. So began the greatest anti-science campaign of the twentieth century.

The dictator, who declared 1929 the year of the “Great Break with the Past,” gave Vavilov an ultimatum: he had to breed his miracle plants in three years, or face grave consequences. It was a biological impossibility; in reality — the evolutionary reality of reproductive cycles and genetic development — it would take at least four times as long for new genetic traits to manifest in a species on the scale of a crop. Seizing upon his spotlight moment and his nascent promotion within Stalin’s scientific establishment, Lysenko launched a concerted attack on Vavilov’s research, pitting it against his own “science” as too slow for the urgently needed famine relief in the country, too humble for the economic domination Stalin craved. He did not hesitate to falsify his own research to bolster its claims.

Vavilov had spent years laboring to bring the seventh International Congress of Genetics to the USSR and although it had been initially approved by the government, now the Communist Party abruptly cancelled the global gathering. When it was eventually convened in Edinburgh after a two-year delay and Vavilov was banned from attending, his international colleagues placed an empty chair on the stage to protest his absence — he was already one of the most respected geneticists in the world.

With science itself under assault, Vavilov devoted all of his energies to his institute and the seed bank, vowing:

We shall go into the pyre, we shall burn, but we shall not retreat from our convictions.

When his plants developed in accordance with nature and failed to meet the dictator’s timeline, Vavilov was accused of treason and sabotage. In the middle of a field expedition in the Ukraine, he was arrested as “an active participant of an anti-Soviet wreckage organization and a spy for foreign intelligence services.” His home was raided and all of his field notes destroyed, but his colleagues managed to save his voluminous correspondence with other scientists and his manuscripts, tucking them away in the basement of the institute, beneath the seed bank.


Nikolai Vavilov’s arrest photo

Upon receiving news of the arrest, Vavilov’s brother wrote in his diary:

His big useful life is being ruined… life of tireless and intense work for his homeland, for the people. All his life spent in work, with no other hobbies. Wasn’t it obvious and clear to everybody? What else can be asked and demanded of individuals? This is a cruel mistake and an injustice. It is even more cruel because it is worse than death. The end of scientific work, the slander, ruining the lives of family members, the threat of it all.

Over the next eleven months in jail, Vavilov was interrogated and tortured hundreds of times, sometimes for thirteen hours a time, for a total of 1,700 hours, with the intention of coercing a confession of sabotage and espionage. He remained adamant that his research had been only in the service of science and human welfare.

Like Dostoyevsky, he was sentenced to death by firing squad, but his death sentence was repealed and reduced to twenty years in a prison camp.

This was an epoch of sweeping terror. While Stalin was terrorizing scientists, Hitler was savaging Europe. Leningrad was next on his conquest list — not only because of its geopolitical advantages as a major international port, but because it housed something precious: the seed bank. The Führer well understood that controlling the world’s food supply was key to controlling the world’s population, so he tasked a special SS unit with looting Vavilov’s seed collections.

On September 8, 1941, the Nazis began their assault on Leningrad by severing the last road to the city. The siege would last 872 days as Leningrad refused to surrender. Food ran out fast. By the winter of 1942, all the government could provide was a ration of two slices of bread, made of 50% sawdust. This too ran out. People took to stripping the wallpaper in their apartments, scraping the adhesive paste made of flour and water, and boiling it to make soup. Death swept the city — 800,000 human beings, one out of every three citizens. Bodies lined the streets unburied. Rats emerged by the millions, feasting on the corpses.

At Vavilov’s institute, scientists barricaded themselves to protect the seed bank from the rats and the Nazis. Famished themselves, they took turns staying up all night, warding off the rodents with metal rods. In what may be the most moving sacrifice in the history of science, nine scientists died of starvation, guarding a cornucopia of nuts, beans, rice, and grains. The curator of legumes was found at his desk, an envelope of peas by his side.

The vault survived unharmed, holding the seeds of life.


Clitoria, or butterfly pea. (Available as a print, a cutting board, and stationery cards, benefitting The Nature Conservancy.)

Meanwhile, Vavilov was languishing in prison. Inmates were fed nothing but flour and frozen cabbage. He survived for two years, his vivacious body shrinking to a skeleton. And then, biology gave way to entropy. In the icy Russian winter of 1943, Nikolai Vavilov died of starvation — the selfsame terror he had devoted his life to preventing. His body was dumped in an unmarked mass grave.

He had once written to a friend:

I really believe deeply in science; it is my life and the purpose of my life. I do not hesitate to give my life even for the smallest bit of science.

Like Alan Turing, Nikolai Vavilov was posthumously pardoned by a new government and eventually celebrated as a hero of science. A Russian postage stamp bears his image and the Russian Academy of Sciences awards a prestigious medal in his honor. A small planet discovered by a Soviet astronomer is named after him, as is a crater on the far side of the Moon. A monument of him rises from a plaza near the prison where he died — a site of frequent resistance protests to this day. The Vavilov Institute of Plant Industry in St. Petersburg is still home to one of the world’s largest seed banks and was the inspiration for the creation of the Svalbard Global Seed Bank near the North Pole in 2008.

When the next global famine savages our species, Vavilov’s legacy will be a lifeline, purchased with his life.

Learning object integration with CMS

Mike's Notes

I made a breakthrough on integrating the CMS and Learning Objects.

Resources

References


Repository

  • Home > Ajabbi Research > Library > Software > Architecture > Learning Object
  • Home > Ajabbi Research > Source > SCORM

Last Updated

19/03/2025

Learning object integration with CMS

By: Mike Peters
On a Sandy Beach: 11/05/2025

Mike is the inventor and architect of Pipi and the founder of Ajabbi.

Yesterday, I finally figured out how to integrate the new Learning Object Engine (lob) with the existing Content Management System Engine (cms). One problem was dealing with these four levels.

  • Simple chunk
  • lesson
  • course
  • set of courses.

I found a solution by examining how a Learning Management System (LMS) created by Brisbane University in 1996 was built around objects and pages. It predates SCORM and has some valuable features.

Now, the interchange with SCORM works, and the CMS creates web pages with learning object types.

The other problem was how to incorporate the Diataxis framework.

"The Diátaxis approach divides documentation into four distinct content types:

  • Tutorials - Lessons that provide a learning experience, taking users step-by-step through hands-on exercises to build skills and familiarity.
  • How-To Guides - Practical guides focused on providing the steps to solve real-world problems.
  • Reference - Technical descriptions and factual information about the system, APIs, parameters, etc.
  • Explanation - Background information and conceptual discussions that provide context and illuminate topics more broadly.

The key premise of Diátaxis is that each content type serves a different user need and has a distinct purpose. Keeping them separated allows the content to be tailored and structured appropriately for that specific goal." - I'd rather be writing.

I created a set of rule combinations that included Diataxis, from which one could be selected. These can drive a Workflow Engine (wfl).

Workflow Example

Select Website > Select Tutorial Section > Create lesson > Save > Create simple chunk > Create content > Save > Post

Testing

The next step is to run test content through both engines and see what happens.

How did places like Bell Labs know how to ask the right questions?

Mike's Notes

I read The Idea Factory: Bell Labs and the Great Age of American Innovation by Jon Gertner some time ago. It was a fascinating insight into a hotbed of discovery and why it was so.

Resources

References

  • The Idea Factory: Bell Labs and the Great Age of American Innovation by Jon Gertner
  • The Bell Telephone Laboratories — an example of an institute of creative technology.

Repository

  • Home > Ajabbi Research > Handbook

Last Updated

18/03/2025

How did places like Bell Labs know how to ask the right questions?

By: Eric Gilliam
FreakTakes: April 21, 2023

Many new science orgs are looking to pursue research that has the positive aspects of both “applied” research and “basic” research. To me, this is a very reasonable approach. After all, the “applied vs. basic research” distinction has always been a rather arbitrary one.

Some research projects feel like they are squarely in one bucket or the other, but it’s not always that clear. Applied research is meant to be research with immediate applications in mind. But, of course, applied research could stumble upon something that leads to a fundamental insight. Basic research is meant to be curiosity-driven research without immediate applications in mind. But, of course, it could quickly lead to a killer application.

In the universe of possible courses of research, there exist many questions that, in the end, will satisfy the spirit of both applied and basic research.

The natural follow-up question is: is finding this subset of golden problems really feasible?

One’s knee-jerk reaction might be that it is not replicable in any kind of systematic way; it is a matter of unreliable personal taste. The history tells a different story. The mid-20th century’s great American R&D labs show us that selecting profitable courses of research that satisfy the spirit of basic research has been done at a high level within large research organizations over the course of several decades.

In this piece, I dive into exactly how Bell Labs ensured that their researchers were working on the right problems. This piece will be the first in a series examining what modern applied research orgs can learn from the great dragons of industrial R&D — places like Bell Labs, GE Research Laboratory, and DuPont’s research department.

Institutions like these not only had Nobel prizes to their names, but each — even though they’ve diminished for various reasons — was quite profitable too. They had their differences, but they all stumbled upon many aspects of managing their research operations that were rather similar. The managers of these organizations — and other researchers at the time — often felt the management decisions they made were common sense rather than some great discovery.

However, these common-sense decisions are things we often don’t do today – but almost certainly should.


Image pulled from a 1922 issue of Bell Telephone Magazine. The image, portraying the switching development department, was used in an article explaining how “systems engineering has played a dominant part in every aspect of Bell Laboratories work” and why it was no surprise that the approach worked equally well in facilitating Bell Lab’s successful World War I work.

Sorry for the delay since my last post. I was 1) working on some projects for some applied science orgs and 2) wrote a piece for the coming issue of Works in Progress that is coming out soon!

You will be getting much more frequent releases from me in the coming couple of months, I promise. As always, if you’d like to discuss how to implement any of the ideas in the piece in your own operation, feel free to reach out on Twitter!

The initial inspiration for much of this piece comes from Jon Gertner’s book, Idea Factory, on the history of Bell Labs. In places, I quote Gertner’s descriptions of events where his words did a better job than mine could.

This piece is done in partnership with the Good Science Project.

Back to the action..

Bell Labs has become legendary in many tech circles. It’s no secret why. Famous ideas and technology like information theory, communications satellites, solar batteries, transistors, and countless other communications-related innovations trace their origins back to Bell Labs in one way or another.

Idolizing Bell Labs for its outcomes is very fair because its outcomes were extraordinary. Many, in recent years, have also begun to idolize Bell Labs’ for its processes. And there’s nothing conceptually wrong with that. If a place has consistently fantastic outcomes and seems to have some secret sauce that is super-additive to the productivity of its researchers, why would we not seek to replicate it?

We should. The issue is that many who idolize Bell’s processes seem to fundamentally misunderstand how the operation worked. The most notable misconception is that many put Bell forward as the poster child of how idle curiosity and the purest kind of basic research can have a role in industry. Looking at the historical sources, that’s not exactly an accurate takeaway.

Bell did a lot of astonishing basic research. But its research, while “basic” for an applied R&D lab, was not nearly as free as many imagine — or, rather, it embodied a different kind of freedom. Equating the freedom of a Fine Hall mathematician at the Princeton Institute for Advanced Study and a physicist at Bell Labs in the 1950s is not an accurate way of looking at it.

Bell was an industrial R&D lab. To an industrial R&D lab, the mission is everything. Frank Jewett, the founding Director of Bell Labs, once said that his new industrial R&D lab was to be:

"An instrument capable of avoiding many of the mistakes of a blind cut-and-try experimentation. It is likewise an instrument which can bring to bear an aggregate of creative force on any particular problem which is infinitely greater than any force which can be conceived of as residing in the intellectual capacity of an individual."

This focus on applications leading the research is not one that faded over the course of Bell Labs’ lifetime.

John Pierce — whose 35-year career (1936-1971) at Bell Labs as researcher and manager encompassed most of Bell Labs’ existence — said this of what made Bell Labs a success:

"Someone depended on them for something, and was anxious to get it. They were really needed, and they rose to the need."

The people who see Bell Labs as a bastion of freedom in private sector research are not entirely mistaken. Bell was pretty damn free for a private-sector lab. It’s just that there was a balancing act.

Jim Fisk was one of the “Young Turks” at Bell Labs — along with Pierce — who helped shepherd in its famous balance of deep research with careful problem selection. He said the following of Bell’s philosophy on problem selection when he was managing Bell Labs:

"Our fundamental belief is that there is no difference between good science and good science relevant to our business. Among a thousand scientific problems, a hundred or so will be interesting, but only one or two will be truly rewarding — both to the world of science and to us. What we try to provide is the atmosphere that will make selecting the one or two in a thousand a matter of individual responsibility and essentially automatic."

This is not Fisk saying that the only relevant problems in electrical communication were those that served Bell’s business interests. But it was him saying that:

  • Many scientific problems are kind of a bore or derivative. He ballparked it arbitrarily at 90%.
  • Some minority of problems are interesting. He ballparked it arbitrarily at 10%.
  • 1%-2% of the interesting problems — that is .1%-.2% of the total problems — would turn out to be worthwhile and relevant to Bell’s work.

The picture this paints of Bell’s preferences for its basic researchers is twofold:

  1. The universe of possible problems is very large. Bell would like its more fundamental researchers to feel free to work on interesting ones.
  2. Among those interesting problems, Bell Labs management would implement systems to make sure researchers identified problems that had a high probability of turning into profitable answers for the Bell Telephone system.

Over the years, Bell Labs management developed a small but coherent set of constraints and rules of thumb to ensure that its researchers internalized that, as Jim Fisk put it, it was “a matter of individual responsibility” to choose the right problems and that Bell Lab’s success in doing this at scale, across thousands of individuals, was “essentially automatic.”

Bell Labs research problem selection: rules of thumb, systems, and constraints

The majority of Bell Labs was made up of applied researchers, development engineers, and other staff — not basic researchers. And these groups usually had a normal boss and projects assigned to them – while Bell’s basic researchers did not. Nevertheless, even though the basic researchers did not have bosses in the traditional sense, they were still nudged to Bell-relevant problems in various ways.

The three key ways Bell Labs nudged its basic researchers toward the right problems were:

  1. Granting researchers what I’ll call a “long leash, but a narrow fence” in which to conduct their explorations.
  2. Facilitating very regular interactions between the basic researchers and Bell’s fundamental development researchers, engineers, manufacturing facilities, and implementation staff.
  3. To top it off, Bell had a corps of what they called systems engineers who ensured that the integration of its best researchers and most pressing problems was not left to chance.

Let’s explore each of these, in turn.

1) Long leash, narrow fence

Bell didn’t exactly tell its basic researchers what they could and could not work on. Not usually at least. A basic researcher’s boss was more of a mentor or advisor than an actual boss.

(Note: the basic researchers made up anywhere from 7% to 18% of Bell Lab's headcount depending on the year and which Young Turk you quote.)

These individuals were guided toward the right problems in other ways. Firstly, it was made clear that the projects should have some obvious bearing on the Bell system and future business. And one was allowed to roam around, so to speak, for a bit before working out exactly what they would be spending their time on, but they should be looking to spend their time on something quite relevant to the business.

The following excerpt from John Pierce’s oral history briefly describes his reflection on his time immediately after joining Bell. He had a particularly high level of freedom in feeling his way around for work, but still found his way into the Bell Labs groove all the same:

"Pierce: I was told to do research on vacuum tubes. People sort of just left me alone. They did suggest that I go and see Philo Farnsworth, who was working on electron multipliers and television pick-up tubes, but I was left pretty much to myself. This was very, very confusing to me. I didn’t know what to do.

Interviewer: Were you doing it alone?

Pierce: Yes

Interviewer: Did they say, “So-and-so has been doing this and this is where he left off”?

Pierce: No. I was just supposed to plan something to do and do it. I think that is close to cruel and unusual punishment."

Pierce, who was giving this interview after his retirement from Bell Labs while he was at CalTech, then continues his reflection:

"Pierce: Too much freedom is horrible. It’s like telling a young child, “Do whatever you want to.” You’ve heard this story. There are various outcomes. One is, “Do I have to do what I want to?” Complete freedom is not very helpful to a person who is inexperienced in the world. It’s certainly bad to be directed to do things very, very narrowly and with no freedom. It’s my guess that for every person who needs more freedom, there are ten people who need more help in finding their way.

Interviewer: So, did they tell you why they wanted the vacuum tubes, when you started off?

Pierce: Not really. I found out some way, inadvertently. Some people were working on electron multipliers, and I made some improvements on them. It became clear that people needed better vacuum tubes for building negative feedback amplifiers, and I worked on that. I don’t think I was told this formally; I just found out by talking to people. Then, as the war approached and we got into war, it became apparent that microwave radar was very, very important, and I worked on tubes for radar. It was a process of osmosis rather than direction that led me into these things, as I remember it."

This Pierce story is an example of things working exactly as they should. An extremely talented young researcher with a background obviously relevant to Bell — multiple electrical engineering degrees from CalTech — came to understand exactly what development work was ongoing at Labs, what it looked like for a basic researcher to be useful to that work, and came up with a course of work to suit those needs.

Morry Tannenbaum, a long-time Bell Labs chemist, famously described this patented level of freedom as “circumscribed freedom.”

Pierce’s story leads us into the second way in which Bell Labs nudged its basic researchers toward the right problems.

2) Frequent interactions with Bell’s development researchers, engineers, manufacturing facilities, and implementation staff

Relationships with the folks who might eventually deploy your research – from those who modified cutting-edge engineering equipment to those that worked in Bell’s factories – ensured researchers were hyper-aware of the problems happening throughout Bell’s massive operation.

The one hard and fast rule Labs did seem to have was that you could not say no to any request for help from any of the applied folks — or other researchers for that matter. Your day-to-day tasks often pertained to your own courses of research, but one’s role as in-house expertise was equally important. These consultations, unsurprisingly, led to all sorts of new ideas for basic research projects.

Another excerpt in Pierce’s oral history is just one of many data points that speaks to the importance of this relationship:

"I remember that during the war we saw a good deal of people from Western Electric [Bell’s manufacturing arm], who were going to manufacture the things that we devised. Because all of these people were engaged in telephony, or during the war because they were all engaged in radar and other military things, you got to talk to people who were engaged in the operation of things, who were engaged in the manufacture of things, and you got a picture of the rest of the world which certainly influenced what research you did."

He continues, diving into how this type of interaction was a natural partner to great basic researchers:

"I can understand a university, which does teaching and research. But the idea of a research institute without ties to either teaching or to manufacturing or operational organization seems a terribly sterile idea. You see that in the Soviet Union; there’s a lot of good activity that never results in anything. When they want to build automobiles, they hire Fiat to build an automobile plant, instead of relying on what they have learned."

(Note: Pierce also believed the university model of research to be an inferior model — for him at least — for reasons I’ll discuss later.)

And, since Bell, as a business and research operation, was far too vast and complex to rely on serendipity to match researchers and problems, it had an entire class of engineers dedicated to ensuring problems and solutions found each other.

3) Bell systems engineers tied this whole process together

Systems engineers – 10% of Bell Labs’ total headcount – were usually technically-trained individuals who spent all of their time, as Jon Gertner put it, keeping:

"One eye on the reservoir of new knowledge and another on the existing phone system and analyzed how to integrate the two. In other words, the systems engineers considered whether new applications were possible, plausible, necessary, and economical."

The existence of systems engineers was Bell acknowledging that a culture of openness and helpfulness was just not sufficient when it came to finding the best problems possible. Of course, implementation/manufacturing/applied research staff often can identify which of their problems are ripe for the research team’s eyes, but a lot of the time they can’t. And yeah, sometimes researchers can come up with great research questions in their area that are ripe for helping a specific group’s work, but a lot of the time they don’t do a great job.

Even if a basic researcher does find a good applied problem, who’s to say that the problem is the best use of their time? There is almost surely a better problem out there. That’s life. The question is, can someone reliably find it?

If a systems engineer does their job well, they can.

Mervin Kelly, long-time Bell Labs manager, described the background of his systems engineers as follows:

"[Systems Engineering] staff members must supply a proper blending of competence and background in each of the three areas that it contacts: research and fundamental development, specific systems and facilities development, and operations. It is, therefore, largely made up of men drawn from these areas who have exhibited unusual talents in analysis and the objectivity so essential to their appraisal responsibility."

Of course, there is a tradeoff here. Instead of using a systems engineer’s skills for normal scientific research or engineering tasks, these individuals were doing other kinds of work. That’s not a small tradeoff. At points, Bell’s systems engineering team was about the same size as, maybe larger than, Bell’s basic research group. But, in a large and complex organization like Bell, this tradeoff was well worth it.

The systems engineers made it their business to be extremely aware of the happenings of the research portions of Bell Labs as well as the minutiae of the industrial portions of Bell Telephone. This included details like:

  • Manufacturing processes to produce electrical parts
  • Which materials or parts tended to degrade and needed to be replaced in the telephone system
  • The cost and time of various repairs and maintenance
  • What portions of Bell’s service were currently bottlenecked by specific technical problems

Of course, no individual systems engineer could know everything and everyone at Bell. But the department as a whole attempted to account for everything, ensuring as little as possible fell through the cracks. And this process worked both ways. These systems engineers, in addition to bringing problems to researchers, also found ways to deploy researchers’ findings in ways that could solve problems in the field or improve Bell’s products and processes.

Mundane systems engineering problem spotting happens when a systems engineer identifies that Bell is spending the equivalent of $1 billion yearly on telephone wire maintenance in certain climates. This engineer can then notify the metallurgy researchers that this problem exists and is begging to be solved.

One Bell Labs executive, a former chemist, loved to point out that a synthetic plastic created by Bell chemists to replace the existing telephone cable sheathing saved Bell “more than the total research budget of Bell Labs for the decade in which the innovation was worked out.” This was an operationally boring problem that could have been hidden in some maintenance budget, away from the eyes of normal researchers. A system engineer exists to identify opportunities just like that.

The ROI of an applied research operation can obviously go up if researchers have problems like this regularly brought to them. And, it should be remembered, these problems were being brought to researchers not just as a veritable gold mine, but with many research-relevant constraints worked out from the beginning. It was the job of the systems researchers to work out as much of this as possible beforehand.

Mervin Kelly believed that the Bell’s continuity was what made it special. If the first two rules of thumb established the continuity, the systems engineers were what ensured it. Kelly spoke about the importance everything connecting the two endpoints of manufacture and basic research in a speech to the Royal Society, saying:

"There has been so much emphasis on industrial research and mass-production methods in my country, that even our well-informed public is not sufficiently aware of the necessary and most important chain of events that lies between the initial step of basic research and the terminal operation of manufacture. In order to stress the continuity of procedures from research to engineering of product into manufacture and to emphasize their real unity, I speak of them as the single entity ‘organized creative technology.’ "

(Please see the bonus piece to learn more about this speech.)

Systems engineers did not technically do anything that you wouldn’t hope would happen with a culture of collaboration, but they were the people that made finding the “one or two in a thousand” problems relevant to Bell and converting them into successful business applications “essentially automatic.”

A Bell systems engineering sketch of Bell’s experimental rollout of a mobile telephone system in Chicago

Freedom comes in many forms

These rules of thumb are contrary to what many believe of Bell Labs’s culture, but most accounts I’ve read from key Labs members point to these methods as a secret sauce that made Bell Labs effective.

The famous stories of Claude Shannon frittering away his days at Bell Labs is a special case. While most researchers did not have the kinds of freedoms Shannon — or many university professors at the time — had, they were usually happy with the tradeoffs. In other ways, many felt life at Bell Labs had a different kind of freedom.

Claude Shannon’s frittering

Stories of Claude Shannon frittering away his time playing games in the Bell lunchroom have become legendary. And I get why. The stories of Shannon building chess machines, juggling, and riding around on unicycles or pogo sticks are great stories. But these stories were not the norm at Bell Labs.

The bulk of these Shannon stories, at least the most “fritter-y” ones, come from after he became a celebrity in the communications world with his discovery — or “invention” depending on your philosophical view of mathematics — of information theory.

On the heels of that discovery, instead of some big financial reward for Shannon, Bell Labs management seems to have paid Shannon with a kind of emeritus status in which they gave the genius the freedom to do whatever he wished. This came with freedom to do unique things like unicycle down hallways or work with the door to his office closed.

Prior to this emeritus lifestyle, Shannon had lived a much more applied, standard life at Labs in which he was often roped into advising the applied folks on their issues and undertook more pressing courses of basic research — fire control and cryptography for the war effort, the mathematics of carrying calls along wires more efficiently, etc.

(Also, it’s worth noting, he used this emeritus-style freedom to accomplish little-to-nothing either for Bell Labs or his personal research agenda in comparison to his earlier life when he was a normal Bell Labs employee or a graduate student on Vannevar Bush’s differential analyzer project.)

In deciding to eventually leave Bell Labs for MIT in 1956, Shannon wrote the following in a letter to his supervisor, Hendrik Bode, on how he was spending his time at Labs:

"It always seemed to me that the freedom I took [at the Labs] was something of a special favor."

He knew that the way he was spending his time at Bell Labs was not in line with its culture. At a university, he felt the freedom he took in terms of focus and hours of work would be less unusual.

Claude Shannon being Claude Shannon

Did the university-trained researchers yearn for the freedom of the university?

The lifestyle of a researcher at Bell Labs, on the face of it, does not seem to have the level of personal freedom university professors had.

Some researchers, like Shannon, did leave to work at universities. Their research often became smaller-scale in terms of resources, but they were more free to do as they wished. A Bell researcher leaving to join a university often viewed it as a lateral move. Shannon wrote in his letter to Bode:

"With regard to personnel, I feel Bell Labs is at least equal in caliber to the general level in academic circles. In some of their specialties Bell Labs is certainly stronger."

University life surely had freedoms that Bell didn’t, but most Bell researchers seemed rather content with the tradeoffs.

Jon Gertner’s book has a great excerpt that recounts what John Pierce thought of the freedoms of his post-Bell home, CalTech, compared to his life at Bell. The excerpt helps one see how, in its own way, the circumscribed freedom of a researcher at Bell was much freer than a professor’s — even in a less bureaucratic era of university life. Gertner writes:

"Pierce went back home to Caltech. For six years he had been doing research and advising graduate students, but he was finding the adjustment difficult. At Bell Labs he had spent his days doing whatever suited him. The brunt of his management work there had consisted of dropping in, unannounced, on colleagues in their labs to ask how work was progressing. But at Caltech he had to give lectures at a prearranged time and then had to spend hours explaining complex ideas to grad students. At Bell Labs, as he recalled it, the same conversation with his colleagues would usually take minutes. (Whether his colleagues actually understood his explanations, or whether he simply walked away before he could field their questions, was a matter for debate.) “I didn’t adapt well to Cal Tech,” he later admitted. “Not that there was anything wrong. For years and years I’d had it too easy. There were very few times when it mattered where I was. I had very few obligations to be at a particular place at a particular time to do a particular thing at Bell Labs.” Pierce obviously seemed to favor the Bell Labs arrangement. As he saw it, the work at the Labs was vital; it was required to improve the network. “People cared about everything,” he said of colleagues there. On the contrary, he noted, in the university “no one can tell a professor what to do, on the one hand. But in any deep sense, nobody cares what he’s doing, either.” "

To Pierce, being genuinely needed was its own kind of freedom.

Karl Compton, who would eventually become President of MIT, wrote a Science article in 1927 dedicated to all the things a university physics department could learn from how industry R&D labs worked at the time. Some of his reasons for supporting this directly address Pierce’s “nobody ever actually depends on you for anything” conundrum. Compton writes:

"Much can…be done to promote cooperation and coordination through actual methods of organization. This has been strikingly demonstrated in some of the big industrial research laboratories, from which the output has greatly exceeded the individual capacities of the research workers and has been achieved only by coordination of effort."

To Compton, the project coordination of places like Bell Labs or GE Research with a clear and limited set of goals — the narrow fence we speak of — was super-additive. The minds and hands, in this setting, added up to far more than they would’ve in a university setting.

It should also be noted that when Compton did eventually take over as the head of a physics department — at Princeton — he was not able to implement any of these lessons. I’m not even sure he tried. Then, as now, changing the structure of an old university was probably a non-starter.

Luckily, newer science organizations like the ones being started today are not so tradition-laden.

John Pierce, in high school, attempting to build his own glider (via CalTech archives)

Teenage John Pierce mid-flight testing one of his gliders. (via CalTech archives)

The mobile phone system: a case study in phenomenal systems engineering

Before concluding, I’d like to paint a picture of what fantastic systems engineering work can look like.

To some, the concept of systems engineers keeping one eye on the reservoir of new knowledge and the other on the details of the phone system does not leave a lot of room for personal glory. Those who feel this way might liken systems engineers to “system quarterbacks” — a mostly derisive word for American football quarterbacks who simply try to facilitate the careful running of the offense instead of attempting to make any big plays themselves.

I don’t think that’s accurate. Great systems engineering work, like great scientific research, has an element of glamor to it. The story of how a trio of Bell systems engineers helped make the mobile phone system a reality should help you see why.

In January 1966, there were rumors that the FCC was thinking about granting Bell Labs access to a larger portion of the limited radio spectrum — a finite resource that the US government decides how to allocate. The range of spectrum being allocated — which twenty years earlier had been allocated to television broadcasters — could be used to make the dream of widespread telephony a reality…probably…somehow.

There were many open questions, but Bell had a kernel of an idea on how to do this. The writeup of the idea was submitted to the FCC by Bell engineers two decades prior. Dick Frenkiel and Phil Porter now found themselves in charge of making the idea a reality. Frenkiel and Porter were both systems engineers at Bell’s rural Holmdel outpost — Frenkiel a mechanical engineer by training and Porter an electrical engineer with a master's in physics.

This was far more exciting work than their previous assignments — at least for Frenkiel who was previously working on machines that could read off pre-recorded messages such as the day’s date. The duo excitedly took to the work.

To start, the major question was, “What would a major course of development even look like for this primitive technology?”

Luckily, they were not starting from scratch. The first step was to dust off Doug Ring and Rae Young’s short 1947 memo to the FCC — two decades before — in which they proposed a non-obvious, very efficient (albeit hypothetical) system in which Bell could use the radio spectrum. Instead of providing coverage to some large circle of users with a cell antenna at the center of it, Ring and Young proposed Bell lay out the system as a honeycomb of hexagons with small antennas at the point of each hexagon and neighboring hexagons could use different frequencies. This would make the limited spectrum allocated to Bell go much further than it otherwise could.

To give the reader an idea, the following images were pulled from Ring and Young’s initial 1947 report. These images show what the layout would look like if the FCC granted Bell either three or four frequencies.


I was not able to figure out if Ring and Young were systems engineers or simply engineers doing systems engineering style work. Regardless, it was fantastic work and a great start to the project. But the most impressive aspects of the project, in my opinion, were almost entirely ahead of this point. After all, the concept of a hexagonal honeycomb is not unfamiliar to many engineers as an efficient way to cover 100% of a space with a circle-like shape that still has vertices.

The honeycomb idea was still a great idea, it just wasn’t going to win anybody any awards on its own. It was the fantastic planning, troubleshooting, and engineering development work of Bell, all started by these two systems engineers, that would turn this idea from a document of eight short pages plus an appendix into a mass-scale engineering reality.

Gertner writes on the (varied) early work of Frenkiel and Porter on this project:

"Neither Frenkiel nor Porter knew precisely how this would be achieved. “It was just two of us,” Frenkiel says. “Nothing important.”

They spent most of 1966 working on the problem — or rather, the problems. The two men covered the walls of their offices with maps and climbed on ladders in various parts of the country to count hills. There were thousands of questions they would need to answer eventually. Many of these were extremely technical, regarding reception and transmission. They talked about signal strength and interference and channel width. They knew every cell would need to be served by what they called “base station” antennas. These antennas would (1) transmit and receive the signals from the mobile phones and (2) feed those signals, by cable, into a switching center that was connected to the nationwide Bell System. Still, several big conceptual problems stood out.

The first was, How large should a hexagonal cell be? Base station antennas would be expensive. How few could they install and still have a high-functioning system?

The second was, How could you “split” a cell? The system would almost certainly start with just a few users—meaning big cells. But as the number of users grew, those cells would subdivide to accommodate the traffic. And more, smaller cells would require more base stations. What was the best and cheapest way?

The third was, How would you “hand off” a call from one cell to another? It had never been done. But it would be the system’s essential characteristic. As a mobile telephone user moved around, how could you switch the call from one antenna to another — from cell to cell, in other words — without causing great distraction to the caller?"

Question by question, they persisted.

A year or so after this work began a third systems engineer, Joel Engel joined the project. Engel — who had a Ph.D. in electrical engineering — was currently assigned to a project on the Bell Boy — a kind of beeper — and was excited to use his spare time on this project because the beeper was largely uninteresting to him. He noted that to get ahead at Bell Labs, “you were supposed to work on more than you were asked to work on.” Still a bit of a newcomer to Bell Labs, he was right on this point. Mervin Kelly used to often tell new hires at Labs, “You get paid for the seven and a half hours a day you put in here, but you get your raises and promotions on what you do in the other sixteen and a half hours.” 

So, Engel joined the duo. The now-trio would huddle into conference rooms to draw hexagons on the blackboard and figure out how the technical pieces of this thing could work.

They were neither true engineers nor business visionaries. The three would surely admit to this second point. They might be more reluctant on the first point — but the proper development engineers would sometimes mention this offhand. The three did not see the true scope of what their project could become. Engel once noted:

"We were not visionaries,” Engel says of the early cellular meetings. “We were techies. If there was a vision it was primarily as a business service. Real estate agents. Doctors who made house calls."

Even as a specialized service for particular businesses, the economics worked. They’d done the math. They’d worked that out along with approximate answers to hundreds of technical questions that needed to be thought through before significant engineering time and resources should be invested in developing the project.

The trio was young and unafraid to work hard, diving into the open-ended, behemoth of a project. The trio’s in-depth planning leveraged:

  • Heavy fieldwork — to understand issues involving the terrain and weather
  • The more conceptual side of their degrees — to understand and extend the initial hexagonal idea
  • (Most of all) Their knowledge of recent developments in various engineering fields — to facilitate the storage and passing of signals

Working out a high-level, implementable, profitable system that could locate a user moving through a honeycomb cell, monitor the strength of the call, and pass the call between channels and towers was the job of the systems engineers. The task required deep scientific, engineering, and operational knowledge of Bell’s installation capacities.

In the end, the trio successfully navigated all of these difficulties. By any measurement, their individual effects on the massive field of mobile telephony are giant — even if their names are not.

Of course, the trio had their limitations. The win was a team win and required the skills of far more than just those three.

While the project did not require any brand-new, Nobel-level academic accomplishments, it did require hundreds of engineering innovations and improvements in the understanding of many pieces of technology. That is where the great development engineers at Bell came in — people like Bill Jakes and Gerry DiPiazza. These were some of “the guys who made cellular real,” as Frenkiel once said.

Jakes was the lead engineer on the fundamental development end of things — this group often carried out less open-ended research projects on things like engineering equipment. Jakes was known to lead crews of engineers out, piling into vans with recording equipment and headphones, to study the effects of forces like obstructions and distances on transmission and reception. They drove thousands of miles, over many months, working through problem after problem related to things like why some particular wave or piece of equipment behaved a certain way in a certain kind of terrain.

As was always the case, the systems engineers were there every step of the way helping coordinate this development work. This was exceedingly necessary as it was not only people like Jakes carrying out this sort of work. It grew into a very large operation — as projects like this always do — across many Bell Labs research locations and teams.

(Check out the bonus piece if you’d like to know more about how this work was coordinated)

One of these research teams operating in parallel, as an example, was led by another great Bell development engineer, Gerry DiPiazza — who I’ve seen called an “engineer’s engineer” in several places. His group was carrying out similar work learning how to build better signal hardware in a stripped trailer home outside of Philadelphia.

DiPiazza reflected on his team and what they were working on when they took their midnight research road trips — when they could test signal strengths and tinker with hardware in a more “noiseless” environment:

"You had to find out, what is the noise level in a suburban environment? How far would a signal go if the antenna was at ten feet, twenty feet, fifty feet? Would it go one mile, two miles, four miles? How many antennas do you need? How do you build an antenna? What are you going to put the antenna on?"

There were a haunting number of problems like the ones DiPiazza describes, worked through by many Bell engineers over many months. And, of course, there’s never any substitute for great fundamental development and engineering work. But, thanks to the likes of Frenkiel, Porter, and Engel, one could be confident that all this money and effort was being spent on the right sub-problems. More importantly, one could be confident that no wicked, unsolvable problems seemed to be awaiting them.

The whole thing had been planned exceedingly diligently. The planners were engineers and Ph.D.s. They already had deep familiarity with the phone system and all its details. They’d brought all the relevant scientific questions to top research minds, consulted engineers who work with relevant tech every day, and coordinated with Bell field staff. This is what they did. When you’re allocating this much money and research time to something, the confidence that systems engineers can provide is invaluable.

They kept a research organization like Bell Labs working on the best problems possible and helped ensure that as few resources as possible were wasted on research that would not turn out to be usable.

The tools of the trade. Equipment testing vans and trailers used by Bell’s development researchers and engineers.

Applied science orgs need a systems engineer

In applied scientific work, great systems engineering work can rival the importance of great research minds.

It was not only Bell that stumbled upon a position like systems engineer to help facilitate its operation. GE Research and Dupont’s research arm — also applied science orgs that left some room for idle curiosity — had similar positions in their own operations that went by other names.

Frankly, it just makes sense. Any new science org attempting to pair usable applied science with some kind of fundamental research should think long and hard before they decide that a full-time systems engineer is not for them.

I’m not saying that, like Bell, a young applied science org needs as many systems engineers as basic researchers. I imagine the ratio of systems engineers to basic researchers should grow as the complexity of either the research operation or the system in which discoveries are going to be applied grows. No new science org has a research operation as large and varied as the mature Bell Labs. So, naturally, the ratio should be smaller.

But many new science orgs do hope to plug into complex systems and operations. Several of the orgs I’ve spoken with have problem scopes that, more or less, pertain to the needs/problems/holes present in all of life sciences research.

Finding a problem in these systems is not so hard for those familiar with the systems. That’s why many researchers and engineers do not feel the need to bring in help. But finding a set of good problems is not finding the best problems. Finding the best problems is a profession in and of itself. A systems engineer is worth it when, under the right scrutiny, it might turn out that the best problem is 10X as financially valuable, does 50X the social good, or is 2X as likely to work as just some run-of-the-mill good problem.

This can be left to researchers. But it shouldn’t.

For every ten or so basic researchers in an applied science org, it feels safe to say there should be at least one systems engineer. Bell Labs had about a 1:1 ratio of basic researchers to systems engineers and a 9:1 ratio of total research, engineering, and facilities development staff to systems engineering staff.

Different orgs will have different needs. But zero is almost surely the wrong number. Even some fraction of one, say 1/4, feels like it’s playing it needlessly cavalierly with such an important piece of an applied research organization. One person who is spending a day or two a week but most of their time on other things feels like a half-measure.

Bell had a massive sample size of engineers, researchers, and problems on their side, and even they didn’t rely on pure probability — serendipity — to do their problem identification for them.

There’s a reason: great problem selection is too important to be left to chance.

Thanks so much for reading! For those interested, I’ve put together a document breaking a speech from longtime Bell Labs leader, Mervin Kelly, on:

  • How Bell Labs was structured
  • What kinds of individuals were hired for each portion of Labs
  • How different sections of Bell Labs were integrated
  • And what day-to-day life looked like for the different roles.

It is great information for those who run their own research operations or are just hardcore hobbyists/massive Bell Labs nerds. Bonus: More detail on how Bell Labs operated

And if you have any interest in being a kind of Bell Labs systems engineer, check out the following post. I am currently working to build more BBNs. Many BBN founders either must act like Bell Labs systems engineers or need to hire someone who can.