Engineering Newsletters

Mikes Notes

I recently discovered and subscribed to a series of software engineering newsletters. Most have a website and often reference other engineering websites. All are written by highly experienced engineers.

Newsletters

Just Use Postgres for Everything

By Stephan Schmidt

Just Use Postgres for Everything

How to reduce complexity and move faster

TLDR; just Postgres for everything.

We have invited complexity through the door. But it will not leave as easily.

There is Radical Simplicity though.

One way to simplify your stack and reduce the moving parts, speed up development, lower the risk and deliver more features in your startup is “Use Postgres for everything”. Postgres can replace—up to millions of users—many backend technologies, Kafka, RabbitMQ, Mongo and Redis among them.

This makes every application easier to develop, scale and operate.

Less moving parts means fewer developers for parts that don’t provide value or just replicate existing functionality (frontend) and more developers on parts (like the backend) that does provide value to customers. What if you could increase feature output by 50% without higher costs? For developers: What about a lower cognitive load? You really deeply understand all moving parts? No more imposter syndrome?

Just Use Postgres

  • Use Postgres for caching instead of Redis with UNLOGGED tables and TEXT as a JSON data type. Use stored procedures or do as I do, use ChatGPT to write them for you, to add and enforce an expiry date for the data just like in Redis.
  • Use Postgres as a message queue with SKIP LOCKED instead of Kafka (if you only need a message queue). Or as a job queue in Go with River
  • Use Postgres with Timescale as a data warehouse.
  • Use Postgres with pg_analytics as an in memory OLAP with Apache Datafusion
  • Use Postgres with JSONB to store Json documents in a database, search and index them - instead of Mongo.
  • Use Postgres as a cron demon to take actions at certain times, like sending mails, with pg_cron adding events to a message queue.
  • Use Postgres for Geospacial queries.
  • Use Postgres for Fulltext Search instead of Elastic.
  • Use Postgres to generate JSON in the database, write no server side code and directly give it to the API.
  • Use Postgres with auditing with pgaudit
  • Use Postgres with a GraphQL adapter to deliver GraphQL if needed.

There I’ve said it, just use Postgres for everything.

Resources

Good code is rarely read

From Alex Molas

Good code is rarely read

06/06/2024

The other day, I was interviewing a developer for a position at Wallapop. The candidate was a bit junior, but I enjoyed their technical assignment, and the conversation was going great. One of the questions I had to ask was

How do you define good code?

I was expecting an answer in the lines of “best practices, DRY, SOLID, code formatting, design patterns, etc.”, but the candidate just answered.

Good code is code that’s easy to read.

And this apparently simple answer got me thinking for the last week. Here are some random thoughts I had.

Why best practices?

Best practices are tools that help us write code that’s easy to read. These practices, such as the DRY principle, following SOLID, and maintaining consistent code formatting, are not arbitrary rules. These guidelines have been developed to ensure that code is understandable and maintainable. When code is easy to read, it’s easier to debug, extend, and refactor. This readability allows other developers (or even your future self) to quickly grasp the logic and purpose of the code without extensive documentation.

Read to write

But no one reads code just for the fun of it 1. When you read code, it’s to understand it and then use it. This means that the purpose of code readability is to facilitate further development. If code is written clearly, it reduces the cognitive load on the developer who will work with it. Code that is easy to read is also code that is easy to use. When functions and classes are named appropriately and their purposes are clear, you can use them without understanding their internal workings.

Code shouldn’t be read more than written

The saying “code is read more than written” is often cited to emphasize the importance of writing readable code. But it could be a symptom of bad code. If a code needs to be read frequently, it might indicate that it is not as clear or intuitive as it should be. Good code should be used more than read. It should be so well-designed that developers can use it without needing to read through it extensively.

Good code is rarely read

Therefore, your goal when writing code is for it to be read as little as possible. This might sound counterintuitive, but if your code is so easy to use that it doesn’t require an in-depth undertanding to be used, then it is good code. It should be so well-structured and named that its purpose and functionality are immediately apparent. This minimizes the need for others to read through and interpret the code, allowing them to use it more effectively.

In conclusion, good code is rarely read. Good code is so ergonomic that you almost don’t need to read it; you just use it. It allows developers to focus on building features and solving problems rather than deciphering existing code. This is the ultimate goal of good coding practices: to create code that is so clear and intuitive that it almost disappears, allowing the functionality to shine through.

CommonMark

From https://commonmark.org/

CommonMark

A strongly defined, highly compatible specification of Markdown

What is Markdown?

It’s a plain text format for writing structured documents, based on formatting conventions from email and usenet.

Who created Markdown?

It was developed in 2004 by John Gruber in collaboration with Aaron Swartz. Gruber wrote the first markdown-to-html converter in Perl, and it soon became widely used in websites. By 2014 there were dozens of implementations in many languages.

Why is CommonMark needed?

John Gruber’s canonical description of Markdown’s syntax does not specify the syntax unambiguously.

In the absence of a spec, early implementers consulted the original Markdown.pl code to resolve these ambiguities. But Markdown.pl was quite buggy, and gave manifestly bad results in many cases, so it was not a satisfactory replacement for a spec. Markdown.pl was last updated December 17th, 2004.

Because there is no unambiguous spec, implementations have diverged considerably over the last 10 years. As a result, users are often surprised to find that a document that renders one way on one system (say, a GitHub wiki) renders differently on another (say, converting to docbook using Pandoc). To make matters worse, because nothing in Markdown counts as a “syntax error,” the divergence often isn’t discovered right away.

There’s no standard test suite for Markdown; MDTest is the closest thing we have. The only way to resolve Markdown ambiguities and inconsistencies is Babelmark, which compares the output of 20+ implementations of Markdown against each other to see if a consensus emerges.

We propose a standard, unambiguous syntax specification for Markdown, along with a suite of comprehensive tests to validate Markdown implementations against this specification. We believe this is necessary, even essential, for the future of Markdown.

That’s what we call CommonMark.

Who are you today?

We’re a group of Markdown fans continually working toward the vision of CommonMark — a standard, interoperable and testable version of Markdown.

  • John MacFarlane, jgm@berkeley.edu
  • Martin Woodward, martinwoodward@github.com
  • Jeff Atwood, jatwood@codinghorror.com

Who were you in 2014, when this started?

We’re a group of Markdown fans who either work at companies with industrial scale deployments of Markdown, have written Markdown parsers, have extensive experience supporting Markdown with end users – or all of the above.

  • John MacFarlane, of Pandoc
  • David Greenspan, of Meteor
  • Vicent Marti, of GitHub
  • Neil Williams, of Reddit
  • Benjamin Dumke-von der Ehe, of Stack Overflow / Stack Exchange
  • Jeff Atwood, of Discourse

How can I help?

Exercise our reference implementations, or find a community implementation in your preferred environment or language. Provide feedback!

If a CommonMark implementation does not already exist in your preferred environment or language, try implementing your own CommonMark parser. One of our major goals is to strongly specify Markdown, and to eliminate the many old inconsistencies and ambiguities that made using Markdown so difficult. Did we succeed?

Where can I find it?

spec.commonmark.org

The CommonMark specification.

code.commonmark.org

Reference implementation and validation test suite on GitHub.

talk.commonmark.org

Public discussion area and mailing list via Discourse.

commonmark.org/help

Quick reference card and interactive tutorial for learning Markdown.

spec.commonmark.org/dingus/

Live testing tool powered by the reference implementation.

When is the spec final?

The current version of the CommonMark spec is quite robust after many years of public feedback.

There are currently CommonMark implementations for dozens of programming languages, and the following sites and projects have adopted CommonMark:

  • Discourse
  • GitHub
  • GitLab
  • Reddit
  • Qt
  • Stack Overflow / Stack Exchange
  • Swift

Payment Service Providers

Mikes Notes

Eventually, Ajabbi will need the services of a payment service provider.

"A payment service provider (PSP) is a third-party company that allows businesses to accept electronic payments, such as credit card and debit card payments. PSPs act as intermediaries between those who make payments, i.e. consumers, and those who accept them, i.e. retailers."

"They often provide merchant services and act as a payment gateway or payment processor for e-commerce and brick-and-mortar businesses. They may also offer risk management services for card and bank-based payments, transaction payment matching, digital wallets, reporting, fund remittance, currency exchange and fraud protection. The PSP typically provides software to integrate with e-commerce websites or point of sale systems." - Wikipedia.

Resources

Using CloudConvert

Mikes Notes

CloudConvert is an excellent file conversion SaaS service. This is the best service available, its fast, and I use the free version most days. Free means less than 25 conversions per day. I will eventually get the paid version. There is an API.

Description

"CloudConvert is an online file converter. We support nearly all audio, video, document, ebook, archive, image, spreadsheet, and presentation formats. To get started, use the button below and select files to convert from your computer." - CloudConvert

"CloudConvert was founded 2012 with the vision to build a universal tool for file conversions. Our product is used both by end users and corporate customers via our API. CloudConvert is built by Lunaweb GmbH, a company located in Munich, Germany.

We believe that it is important to focus on core business competencies and let others do everything else. This is why we are doing what we are best at doing: converting files.

We believe that data security is the most important good in our industry. This is why data protection and security have the highest priorities for us.

We believe in small, agile and unbureaucratic teams."

Pricing

"You can use CloudConvert absolutely free for up to 25 conversions per day. Beyond that we offer flexible payment options. Use the quantity slider below to customize a package or subscription according to your needs." - CloudConvert

Universally Unique Identifier (UUID)

"A Universally Unique Identifier (UUID) is a 128-bit label used for information in computer systems. The term Globally Unique Identifier (GUID) is also used, mostly in Microsoft systems. [1][2]

When generated according to the standard methods, UUIDs are, for practical purposes, unique. Their uniqueness does not depend on a central registration authority or coordination between the parties generating them, unlike most other numbering schemes. While the probability that a UUID will be duplicated is not zero, it is generally considered close enough to zero to be negligible.[3][4]

Thus, anyone can create a UUID and use it to identify something with near certainty that the identifier does not duplicatAe one that has already been, or will be, created to identify something else. Information labeled with UUIDs by independent parties can therefore be later combined into a single database or transmitted on the same channel, with a negligible probability of duplication.

Adoption of UUIDs is widespread, with many computing platforms providing support for generating them and for parsing their textual representation." - Wikipedia

Versions

The OSF DCE variant defines eight "versions" in the standard, and each version may be more appropriate than the others in specific use cases.

  • Versions 1 and 6 (date-time and MAC address)
  • Version 2 (date-time and MAC address, DCE security version)
  • Versions 3 and 5 (namespace name-based)
  • Version 4 (random)
  • Version 7 (timestamp, counter and random)
  • Version 8 (custom)

Collisions

For example, the number of random version-4 UUIDs which need to be generated in order to have a 50% probability of at least one collision is 2.71 quintillion, computed as follows:



This number is equivalent to generating 1 billion UUIDs per second for about 86 years. A file containing this many UUIDs, at 16 bytes per  UUID, would be about 45 exabytes.

The smallest number of version-4 UUIDs which must be generated for the probability of finding a collision to be p is approximated by the formula

Thus, the probability to find a duplicate within 103 trillion version-4 UUIDs is one in a billion.

Issues

Collisions have occurred when manufacturers assign a default UUID to a product, such as a motherboard, and then fail to over-write the default UUID later in the manufacturing process. For example, UUID 03000200-0400-0500-0006-000700080009 occurs on many different units of Gigabyte-branded motherboards.

Common Design Patterns at Stripe

Mikes Notes

An excellent series of articles on designing APIs from Stripe, The Payment Service Provider. There are detailed examples and coding conventions.

Designing APIs for Humans Series' Articles by Paul Asjes (Stripe Dev)

Examples

$pi = $stripe->paymentIntents->create([
  'amount' => 1000,
  'currency' => 'usd',
  // Without prefixes, we'd have to supply a 'type'
  'payment_method' => [
    'type' => 'card',
    'id' => '1LaRQ7GUcADgqoEMV11wEUxU'
  ],
]);

Docs

Stripe also has superb Developer Docs, which are a helpful format to start with when designing Pipi Docs, especially for the API.

Resources

Other stuff

Dashboard Design Patterns

Mikes Notes

Vitaly Friedman, editor of Smashing Magazine, recently posted resources about Dashboard Design Patterns.

"This page lists design patterns for dashboard design collected to support the design and creative exploration of dashboard design. We run a dedicated workshop in March 2022 to help you applying and discussing design patterns in your work.

The patterns are coming out of our research, described in our publication:

Bach, Euan Freeman, Alfie Abdul-Rahman, Cagatay Turkay, Saiful Khan, Yulei Fan, Min Chen: Dashboard Design Patterns, IEEE VIS Conference / Journal of Transactions on Visualization and Computer Graphics (TVCG), 2023. .

What are Dashboards?

Dashboards offer a curated lens through which people view large and complex data sets at a glance. They combine visual representations and other graphical embellishments to provide layers of abstraction and simplification for numerous related data points, so that dashboard viewers get an overview of the most important or relevant information, in a time efficient way. Their ability to provide insight at a glance has led to dashboards being widely used across many application domains, such as business, nursing and hospitals, public health, learning analytics, urban analytics, personal analytics, energy and more." - https://dashboarddesignpatterns.github.io/


Resources

Theorem Provers

Mikes Notes

I have a fortnightly online meeting with my computer scientist friends Alex and Chris to discuss our various computer and mathematics projects. It is an excellent way of getting feedback and being exposed to new ideas.

Earlier this year, Alex mentioned an online talk at the London Mathematical Society that he attended. I then watched it. It was about "proving assistants," AKA "theorem provers."

I find mathematics papers unfathomable. But I could follow along quite easily reading the proofs generated by Isabella, the proving assistant.

"Automated theorem proving (also known as ATP or automated deduction) is a subfield of automated reasoning and mathematical logic dealing with proving mathematical theorems by computer programs. Automated reasoning over mathematical proof was a major impetus for the development of computer science." - Wikipedia.

Isabella

"Abstract: The formalisation of mathematics is an ongoing process that arguably started as early as the 19th century, intensified with the foundational crisis at the start of the 20th century, and since the 1970s has been conducted with the help of computers. Recent decades have seen the machine formalisation of lengthy and technically complicated proofs, but some have argued that even these were not representative of modern mathematics. Recent achievements by a number of different groups are starting to challenge this scepticism. The speaker will outline some of these, while also noting some of the remaining trouble spots.

This was the LMS-BCS FACS Seminar, which took place on Monday 15 January 2024 online via Zoom." - LMS YouTube Channel

Vampire

"Abstract: Vampire is a fully automated theorem prover for first-order logic that has been developed for over 25 years. It has a long tradition of being the “best” in the world (solves the most in the least time at the theorem prover competition) for traditional first-order logic problems found in many domains. However, this traditional first-order logic setting often struggles to capture problems from program verification that need to talk about theories such as arithmetic. Indeed, program verification is typically the domain of SMT solvers (Boolean SAT solvers extended with theory decision procedures). However, we often see problems in program verification that combine theories and quantification (the bread-and-butter of traditional first-order logic reasoning) where SMT solvers struggle with heuristic approaches. Therefore, around 10 years ago we embarked on a journey to extend Vampire to the program verification space, adding theory reasoning, induction, interpolation, and a some other fun things. It is this journey, the challenges, advances, and applications, that I will talk about in this presentation

Part of the LMS Computer Science Colloquium 2023, which took place on Friday 1 December 2023 at De Morgan House, London and online via Zoom." - LMS YouTube Channel

Resources

Example in Isabella

For example, a declarative proof by contradiction in Isar that the square root of two is not rational can be written as follows.
theorem sqrt2_not_rational:
  "sqrt 2 ∉ ℚ"
proof
  let ?x = "sqrt 2"
  assume "?x ∈ ℚ"
  then obtain m n :: nat where
    sqrt_rat: "¦?x¦ = m / n" and lowest_terms: "coprime m n"
    by (rule Rats_abs_nat_div_natE)
  hence "m^2 = ?x^2 * n^2" by (auto simp add: power2_eq_square)
  hence eq: "m^2 = 2 * n^2" using of_nat_eq_iff power2_eq_square by fastforce
  hence "2 dvd m^2" by simp
  hence "2 dvd m" by simp
  have "2 dvd n" proof -
    from ‹2 dvd m› obtain k where "m = 2 * k" ..
    with eq have "2 * n^2 = 2^2 * k^2" by simp
    hence "2 dvd n^2" by simp
    thus "2 dvd n" by simp
  qed
  with ‹2 dvd m› have "2 dvd gcd m n" by (rule gcd_greatest)
  with lowest_terms have "2 dvd 1" by simp
  thus False using odd_one by blast
qed

The Stanford Encyclopedia of Philosophy

Mikes Notes

The Stanford Encyclopedia of Philosophy (SEP) might be a helpful resource. Some science papers I have to read to build Pipi are very academic. 

"Welcome to the Stanford Encyclopedia of Philosophy (SEP), which as of Summer 2023, has nearly 1800 entries online. From its inception, the SEP was designed so that each entry is maintained and kept up-to-date by an expert or group of experts in the field. All entries and substantive updates are refereed by the members of a distinguished Editorial Board before they are made public. Consequently, our dynamic reference work maintains academic standards while evolving and adapting in response to new research. You can cite fixed editions that are created on a quarterly basis and stored in our Archives (every entry contains a link to its complete archival history, identifying the fixed edition the reader should cite). The Table of Contents lists entries that are published or assigned. The Projected Table of Contents also lists entries currently unassigned but nevertheless projected." - SEP

Resources

How Signiant documents their API

Mikes Notes

I found this API documentation page. It is generated by Swagger, and it looks great.

"Signiant Flight Deck provides a REST API to automate user management, user group, agent groups, jobs, resource controls, and job groups through the Manager software." - Signiant

List of API Endpoints


API Endpoint Example


The code

[
  {
    "user": {
      "userName": "exampleuser",
      "fields": {
        "accessAcls": [
          {
            "edit": true,
            "execute": true,
            "read": false,
            "remove": true,
            "userName": "admin"
          }
        ],
        "cell": "1-555-555-1234",
        "email": "user@example.com",
        "failedLoginTimePeriod": "24",
        "fax": "1-555-555-4321",
        "firstName": "Example",
        "groups": "exampleGroup",
        "ignoreAdministrationForSoapAuth": false,
        "lastName": "User",
        "maxAllowedLoginAttempts": "10",
        "menus": [
          {
            "name": "Groups",
            "parent": {
              "name": "Jobs"
            }
          }
        ],
        "organization": "Example Organization",
        "password": "aValidPassword1",
        "phone": "1-555-555-1111",
        "roles": {
          "isTemporary": false,
          "isAdminGuiLogin": false,
          "validFrom": null,
          "validTo": null,
          "isSysAdmin": false,
          "isOrgAdmin": false,
          "isWorkflowAdmin": false,
          "isWorkflowSupervisor": false,
          "isCompEditor": true,
          "isMonitorUser": false
        },
        "title": "Dr."
      }
    }
  }
]

Mikes Notes

For security, use both

  • API key
  • Access tokens from an authorisation server

Web app manifests

Mikes Notes

I recently used Favicon.io, an online app to create the favicons for the Ajabbi websites. One of the generated files was called site.webmanifest, which I had never heard of before. Everything below comes from MDN.

This is not in production.

Notes

"A web application manifest, defined in the Web Application Manifest specification, is a JSON text file that provides information about a web application.

The most common use for a web application manifest is to provide information, such as the app's name and icon, that the browser needs to install a progressive web app (PWA) on a device.

A web application manifest contains a single JSON object with the top-level keys called members." - MDN.

Example

JSON
{
  "name": "HackerWeb",
  "short_name": "HackerWeb",
  "start_url": ".",
  "display": "standalone",
  "background_color": "#fff",
  "description": "A readable Hacker News app.",
  "icons": [
    {
      "src": "images/touch/homescreen48.png",
      "sizes": "48x48",
      "type": "image/png"
    },
    {
      "src": "images/touch/homescreen72.png",
      "sizes": "72x72",
      "type": "image/png"
    },
    {
      "src": "images/touch/homescreen96.png",
      "sizes": "96x96",
      "type": "image/png"
    },
    {
      "src": "images/touch/homescreen144.png",
      "sizes": "144x144",
      "type": "image/png"
    },
    {
      "src": "images/touch/homescreen168.png",
      "sizes": "168x168",
      "type": "image/png"
    },
    {
      "src": "images/touch/homescreen192.png",
      "sizes": "192x192",
      "type": "image/png"
    }
  ],
  "related_applications": [
    {
      "platform": "play",
      "url": "https://play.google.com/store/apps/details?id=cheeaun.hackerweb"
    }
  ]
}

Deployment of Manifest

Web app manifests are deployed in your HTML pages using a <link> element in the <head> of a document:

HTML

  <link rel="manifest" href="manifest.json" />


The .webmanifest extension is specified in the Media type registration section of the specification (the response of the manifest file should return Content-Type: application/manifest+json). Browsers generally support manifests with other appropriate extensions like .json

Content-Type: application/json

If the manifest requires credentials to fetch, the cross-origin attribute must be set to use credentials, even if the manifest file is in the exact origin as the current page.

HTML

<link rel="manifest" href="/app.webmanifest" crossorigin="use-credentials" />

Resources

How Is Science Even Possible?

From The Joy of Why

https://www.quantamagazine.org/how-is-science-even-possible-20240620

  • The link above has a podcast player link for the audio

How are scientists able to crack fundamental questions about nature and life? How does math make the complex cosmos understandable? In this episode, the physicist Nigel Goldenfeld and co-host Steven Strogatz explore the deep foundations of the scientific process.

The universe seems like it should be unfathomably complex. How then is science able to crack fundamental questions about nature and life? Scientists and philosophers alike have often commented on the “unreasonable” success of mathematics at describing the universe. That success has helped science probe some profound mysteries — but as the physicist Nigel Goldenfeld points out, it also helps that the “hard” physical sciences, where this progress is most evident, are in major ways simpler than the “soft” biological sciences.

In this episode, Goldenfeld speaks with co-host Steven Strogatz about the scientific importance of asking the right questions at the right time. They also discuss the mysterious effects of “emergence,” the phenomenon that allows new properties to arise in systems at different scales, imposing unexpected order on cosmic complexity.

Transcript

STEVEN STROGATZ: Albert Einstein once wrote, “The eternal mystery of the world is its comprehensibility.” It really is awesome when you think about it. The laws of nature, at least in physics, turn out to be amazingly simple. So simple that we human beings can discover those laws and understand them and use them to change the world.

But why is nature like this? Why is it so comprehensible? And why is math so uncannily effective at explaining it, not just in physics, but also in chemistry, in astronomy, and even in some parts of biology? In short, why is science even possible?

I’m Steve Strogatz and this is “The Joy of Why,” a podcast from Quanta Magazine, where my co-host, Janna Levin, and I take turns exploring some of the biggest mysteries in math and science today. In this episode, we’ll be speaking with physicist Nigel Goldenfeld about the mystery of nature’s comprehensibility.

Nigel holds the Chancellor’s Distinguished Professorship in Physics at the University of California, San Diego, where his research spans condensed matter theory, the theory of living systems, hydrodynamics and non-equilibrium statistical mechanics. Previously, he was a professor at the University of Illinois at Urbana-Champaign, and a founding member of its Institute for Genomic Biology, where he led the biocomplexity group and directed the NASA Astrobiology Institute for Universal Biology. In addition to being a fellow of the American Physical Society, the American Academy of Arts and Sciences, and the U.S. National Academy of Sciences, Nigel is also well known for authoring one of the standard — and I have to say, terrific — graduate textbooks in statistical mechanics.

Nigel, thanks so much for coming on the show.

NIGEL GOLDENFELD: Oh, it’s a pleasure to be here, Steve.

STROGATZ: Yes, it really is a pleasure for me. I am curious where we’re going to go with this. It’s such a really very profound philosophical question, this Einstein quote about nature’s comprehensibility, but I wonder what you think of it? I mean, let’s talk about both parts of it. Is the world really comprehensible, at least to some degree? And if it is, does that strike you as mysterious?

GOLDENFELD: So I think it’s a wonderful quote, and certainly one that inspired me, and I’m sure other people thinking about the research that we do. And I think the reason it’s important is because we’ve grown physics to such an extent that it now starts to impinge on other disciplines. You mentioned biology, but also, you know, I could mention economics and atmospheric sciences, climate change, all these sorts of things.

And as you start getting into these much more complex and complicated areas of science, you wonder how were we even able to do anything in physics, let alone these other things. And in fact, the reason these other fields are difficult is something that’s also not clear. You know, you could also ask what is the reason for the unreasonable ineffectiveness of mathematics in biology.

[STROGATZ laughs]

GOLDENFELD: When you start to think about it, you realize that when we talk about the effectiveness, we’re talking about problems where we’ve been lucky to make an impact. And so our sample is skewed.

We have a lot of successes in science. Some of the most accurate things that we know in science are in physics. You could say, “Well, that’s because, you know, we only talk about those problems, because those are the ones that actually worked. All the many other things that we try to do failed dismally, and we never ask about those. And our sample is somewhat biased.”

STROGATZ: Well, that’s great, this point that you’re making that we’re sort of assuming facts not necessarily in evidence here in saying that the world is comprehensible. Because as you say, there are these parts of science that we still have yet to really figure out — economics, parts of atmospheric science and so on.

So for listeners who aren’t necessarily following what we’re talking about here, think about the example from the 1850s or ’60s: James Clerk Maxwell figuring out the equations for how electricity and magnetism work.

It’s just four little equations that nowadays fit on a T-shirt — physics and math nerds like me and Nigel and maybe even you like those T-shirts. What’s crazy is that you can really understand almost everything there is to know about electricity and magnetism with the help of those equations and some clever math.

For instance, Maxwell himself figured out that a prediction from those equations is that something called electromagnetic waves could exist. And today, those are the basis for wireless communication, technology that we all use every day in our cell phones.

And so the question is: How is it possible that we, with our puny primate brains, can figure out these four equations that are so marvelous? And is it, as you suggested, Nigel, just that we’re asking questions whose answers are likely to be simple and ignoring the really hard ones? Or, I don’t know, how should we think about this? How is it possible Maxwell could have come up with these equations?

GOLDENFELD: Let me take another example, which is Einstein’s prediction of gravitational waves — which has been in the news a lot in the last couple of years. The story is that Einstein had this idea of thinking about somebody falling in an elevator. And they realized that the falling in an elevator is similar to what you get from gravity.

And so they came up with a principle of equivalence. And from that very, very slender insight, translated into mathematics through Riemannian geometry and tensor calculus and so on, which Einstein had to learn in order to do that, he was able to create this amazing mathematical edifice, which we call the general theory of relativity today, which is actually the theory of gravitation.

And it explains gravitation to a higher accuracy than Newton’s law of gravitation, and makes numerous predictions, of which the gravitational waves are one of the most spectacular. So that’s another fantastic example.

And it just boggles the mind that somebody could imagine that and create the science that makes these predictions. And, you know, a hundred years later using astonishing technology, we’re able to actually observe these things.

STROGATZ: It is. It seems almost like a miracle. It’s something that the physicist Eugene Wigner, in a famous essay in 1960, posed [as] “the unreasonable effectiveness of mathematics in the natural sciences,” and you already alluded to this phrase of his. What is unreasonable about it?

GOLDENFELD: So you and I have been talking about new qualitative phenomena that you predict, for example, from Faraday’s law and all these things that Maxwell had to work with. That’s one thing that’s very important about science, is that we can predict things that you would otherwise not expect.

But the unreasonable effectiveness that Wigner is talking about is the accuracy with which it makes those predictions.

So here’s another example. You look at, say, the quantum mechanics of an atom interacting with Maxwell’s electromagnetic field. When you take electromagnetic field, you apply quantum mechanics to the interaction of that with an atom. You’re able to make predictions to something like 10 decimal places of accuracy. And those agree with experiments to all significant figures that the experiments, in theory, are applicable for. And that’s astonishing. And I think Wigner and Einstein wanted to know how could it be that such very simple mathematics has such great explanatory power.

And people may say, ”Well, what do you mean it’s simple? You know, Einstein’s theory of relativity, general theory of relativity, is one of the most complicated pieces of mathematical physics that you can learn.” And that’s true.

But the physical insight that goes into it is literally very simple. Just, acceleration is literally like a gravitational force. And then being able to turn that into a mathematical equation, which you can then make simple predictions from, is really where the beauty and the amazingness lies. So, I think that’s one aspect of it.

There’s another thing, though, that is not talked about very much, which is that this idea that mathematics and physics is so powerful in its explanations makes another assumption. That assumption is reductionism.

This goes back to another quote of another founder of modern physics, Paul Dirac, who wrote down the relativistic wave equation, an equation that describes quantum mechanics connected with special relativity — it’s called the Dirac equation. And he rather arrogantly wrote that his equation describes most of physics and all of chemistry.

[STROGATZ laughs]

GOLDENFELD: And his idea is that basically — and it’s the same idea that motivates what today we call high-energy physics, but in an era with more bravado would be called elementary particle physics.

And there was the idea that you can just find the elementary building blocks of matter. And then once you’ve got those, all you have to do is put them together and you’ve explained everything in the world. And we know that that’s not true. And that’s the sort of fundamental insight that came out of physics around about 1950 or so. Led to the birth of what’s known as condensed matter physics, and is certainly operative on steroids when you look at biological phenomena, where just knowing the basic forces between atoms doesn’t explain, you know, why you can think.

So when we talk about the effectiveness, we’re talking about the effectiveness on very simple problems.

STROGATZ: Hmm, interesting distinction. So just to review some of these examples again to make sure I’m with you. With Maxwell, his equations, not so simple unless you know vector calculus or something equivalent. But then once you know that math, as I tried to emphasize, it’s just four little equations that can fit on a T-shirt. So simple in that way, and simple principles going into them.

But then your point seems to be, yes, but you can only predict simple phenomena like a propagating wave through a vacuum, whereas really complicated stuff, say, predicting patterns of thought in a human mind — I mean, this is the tricky part.

In principle, do we believe that it is actually somehow in the physics, but we just can’t figure out how to do the math to show phenomena like consciousness and emotion and all that? Or is there something else than what the physical laws imply?

GOLDENFELD: Well, I think there is. And this goes back to the question of why it is that we can do science at all.

If you truly believe that to understand, say, the phenomenon that we see in biology, you can get all of that, say, from Dirac’s equation or, you know, quantum mechanics and so on, then every time you try to understand something quantitatively — in biology or solid state physics, for example — you know, you’d have to worry about the radiative corrections to the mass of the top quark. And none of us think that all of those things that happen at such small scales inside a nucleon at very high energies have anything to do with, you know, why a bird can fly or stuff like that.

The fact that we can do science tells us that somehow these scales get separated through something which we typically call emergence. The great benefit of that is that we don’t have to solve everything all the way down in order to understand something.

STROGATZ: Interesting. So you’re saying worrying about quarks isn’t going to tell us anything about the behavior of the stock market tomorrow. We can somehow… It’s like, as if different scales in nature are insulated from each other, or something like that. What’s the language you would use? You spoke of separation.

GOLDENFELD: I talked of separation and I talked about emergence. And I’d like to give you another example of that which is very different from the one that people like Einstein and Wigner and Dirac and so on would’ve used, and they wouldn’t even have known about it.

So there’s a phenomenon in nature called a phase transition. The simple example is, you take a lump of ice and heat it up, and eventually it’ll melt into liquid. So it’ll go from the solid phase into the liquid phase. And then from there, if you heat it up further, it’ll go into the gas phase.

And another example would be if I took a magnet and I heated it up. It turns out that above a certain temperature, a magnet will stop being magnetic. There is a theory of that transition, the magnetic transition, and other transitions which are like it, such as how materials become superconducting and very exotic things like that. But the most interesting thing about the transition is that we can understand it using a branch of physics called “renormalization group theory.” And I’m not going to go into the technicalities of it, but what the theory predicts is that if you measure how magnetic something is very close to the temperature where it first becomes a magnet, whilst also applying a magnetic field, you get a certain magnetization that you can measure as a function of temperature and external magnetic field.

And you can do this for any magnet that you’d like. But it doesn’t really matter what the atoms are. And if you take the data and process it in a certain way, what you find is the results are the same for every single magnetic material. It doesn’t matter what it is. As you go just below the temperature where it first becomes magnetic, you find that it obeys a certain equation. And that equation is exactly the same for every material. And not just exactly the same: All the data from all the different magnetic materials, they all lie on one curve. And physicists call this universality. We completely understand that.

Now, the other thing is amazing, though, is that we can make a theoretical prediction about what that curve should be if you process the data in the way that the theory tells you to do it. And when you take the data and you take the theoretical curve, it falls exactly on the experimental data.

OK, so that’s fantastic. This model of what a phase transition is is very successful and obviously extremely accurate. Not only does it predict this universality, but it also predicts exactly not just a number, but a whole function.

And it’s a whole relationship that you can measure experimentally. So that’s true. And I like to say that it’s not really true that the model has given a precise prediction in agreement with experiment. It’s really a model of a model of a model of a model.

STROGATZ: [laughs] OK, what?

GOLDENFELD: Yeah. OK.

STROGATZ: You better explain that.

GOLDENFELD: Yes. A model of a model of a model of a model. So, so why is that? Well, so suppose you said to a scientist, OK, make a theory for me of a magnet. So they’d say, well, a magnet is made out of atoms. So in order to understand atoms and how they interact and become magnetic, I need to worry about the electrons on the atoms. I need to worry about the magnetic moments of those atoms. And so I make a model of the material based on quantum chemistry.

But that model is unimaginably complicated, and it gives you no hint that there could be something that doesn’t depend on atoms in it. Because the model itself is very specific to the particular atoms.

So then you say, well, really, that’s way too hard. Maybe a quantum chemist could simulate this and make a prediction. And if they did that, they would see that the prediction did agree with what you see experimentally, and does agree with what the theory predicts. But that’s a very huge computer calculation.

So then you say, let’s simplify it. Let’s just not worry about the atoms too much. Let’s just worry about how the electrons move around in the material. So you go ahead and do that, and you find you’ve got a complicated model of electronic structure.

STROGATZ: Sorry let me interrupt for a second, just to make sure that this whole model of a model thing is clear. So there was the real magnet, then there was the quantum chemistry model of the magnet, then there was the electronic structure model of the quantum chemistry model.

GOLDENFELD: Yes, well, now we’re going to go to the quantum Heisenberg model of the magnetic moments of the electrons inside the electronic structure, which came from the quantum chemistry. And that model is too hard.

So you say, OK, well, let’s throw away quantum mechanics. We’ll just make it classical. So you do that, and the model is still too complicated. So then you say, well, let’s take the thermodynamics, which is what everything depends upon in any case, and let’s do some kind of expansion of that. And that’s a model where you can finally do a calculation.

As you said, you’ve got one, two, three, four, five models of a model of a model of a model of a model of this material. And at each step along the way, you have made an approximation that would be rejected from every physics journal. Because everybody would say, “That’s approximation you can’t justify. There’s no small quantity. No idea what you’re talking about. How can that possibly work?”

STROGATZ: I must also say that here in the math department, you know, people would be hysterical

GOLDENFELD: Oh, yes. Oh, yes. They would be horrified. But the joke’s on them. Because, at the end of the day, you do this whole procedure, and then you find you make a prediction with no adjustable parameters, and it agrees precisely with experiment.

STROGATZ: Dun, dun, dun. [laughs]

GOLDENFELD: Dun, dun, dun. Every step along the way, the approximations you’re making are not systematic and not justifiable, at least ahead of time. And that, I think, is a fantastic way to articulate this mystery that you’re alluding to.

STROGATZ: Hmm, that is a marvelous exposition. I didn’t imagine this ahead of time while preparing for this interview, but I love it. And I think you’re really capturing the mystery. It’s like we have no right for this to work as well as it does. It’s as if nature is somehow acting in a very forgiving or convenient or cooperative manner for us. Like it’s helping us get lucky or something.

GOLDENFELD: Well, that’s the thing. This happens only under special circumstances. In this particular case, very close to a phase transition. So we understand how it works there. But these different levels of description that I alluded to, you know, all of these are different ways of describing something at different length and time and space scales. And as you go to each level, you kind of absorb all the complications of the level lower down into some parameter that is in the description that you’re talking about. And then once you’ve done that, you don’t need to worry about what happened below.

That, I think, is why we can solve this particular problem and why it works so accurately.

STROGATZ: We’re going to take a short break and we’ll be right back.

STROGATZ: Alright, welcome back. I’m speaking with Nigel Goldenfeld about how we can model complex phenomena — and how we manage to do it so accurately.

GOLDENFELD: So, when we talk about how we can do science at all, here is an example which says the only reason you can do this sort of calculation is because there’s these separations of scales and energy and time and space.

When you start talking about, you know, physics being successful, and biology or economics or social interactions and things like that. Can we expect to be able to do those things if there isn’t any obvious way that one can separate scales, and make sure that what happens at very small scales doesn’t affect what happens at large scales? And it may be that there’s some areas of science where that is not true. And then you may not be able to be successful in those things.

STROGATZ: Hmm. That’s an interesting point. I may be going off the rails here, but I’m thinking of something like economics, which you might want to think of as the byproduct of hundreds or thousands or millions of people and firms interacting through markets and so on. That it’s a kind of complex system, economics, where the smaller scale, the molecules or the atoms or the quarks are people making individual decisions that then aggregate into an economy or a market.

In your example, where the fussy behavior of the top quark doesn’t affect what’s happening to the birds flying overhead, here we might not have that separation. Like, individual decision makers can have an outsized impact on the economy? Is that the issue that makes economics so difficult or one of the issues?

GOLDENFELD: Yeah, I don’t know about economics per se, but I’ve given this some thought in terms in finance. So, finance is a very interesting example to think about emergence. So remember in finance, we have data. We know every single transaction that occurred. We know when it occurred, how much. We have every piece of information like that. And now the question is, can you make predictions based on it?

So let me give you an example. First of all, of course, we know that you can’t predict things very well, and not only can you not predict things into the future, you can’t even predict things into the past.

[STROGATZ laughs]

GOLDENFELD: So, there was a wonderful example of this, which was an event called the Flash Crash. Do you remember what that is?

STROGATZ: You should remind us. I’m not sure I remember when and what happened.

GOLDENFELD: On May the 6th, 2010, there was a trillion-dollar crash of the U.S. stock market. The Dow Jones plunged like a thousand points within a few minutes. And eventually it came back up again. And this was an unexpected event, and to this day, people aren’t really 100% sure what triggered that. It certainly wasn’t something that people expected at the time.

What actually happened, I believe, is that you have a cooperative phenomenon where a lot of people are doing algorithmic trading, they’re all more or less using the same signals to trigger their computer guided trades and I think the whole system just synchronized and crashed and eventually people had to stop the thing happening by pulling from the network and things like this. So this is an example of extreme sensitivity cascading through the system because of collective properties of the whole financial system, properties that nobody even knew were there.

STROGATZ: Hmm. Yeah, it’s interesting to hear you use the word “cascade,” because that comes up in connection with the power grid, where sometimes you’ll have an event like a lightning storm somewhere and then because, as you say, there’s this connectivity, in this case, through high voltage transmission lines in the power grid, you can get propagating failures.

So this does seem to be another example where a small-scale event can propagate and have consequences at a much broader scale. So is this the idea why maybe the hard sciences are the easiest?

GOLDENFELD: Oh, I always say that the hard sciences are the easiest. Yes, the reason physics is so successful is because we only ask very simple questions.

STROGATZ: So the supposed soft sciences are, in a certain sense, you would say then, the hardest?

GOLDENFELD: So you have to ask a question. You know, what is the purpose of science? What do we want to be able to predict? So, let’s go back to my example about the phase transition. I talked about this example of how you can look at the behavior of a magnet very close to the temperature where it becomes a magnet. And there’s a universal phenomena there, and we understand it exquisitely, and it’s wonderful and it’s amazing. So, the listener might get the impression that we understand everything about this and there’s nothing mysterious about it at all. But there is still one thing that I didn’t tell you. And that is this.

There is a temperature where every material becomes a magnet. But that temperature is different for each of the materials. And we don’t know how to predict that number very accurately. That number is not something that is universal, unlike the curve that I alluded to that tells you the response of a magnet.

That number depends on everything. All of the levels of description that I swept under the rug in order to explain what happens near a phase transition. All of those things come back to bite you when you want to know what is that actual critical temperature where the material first becomes a magnet.

STROGATZ: Hmm, interesting.

GOLDENFELD: So you have to ask the questions that you ask in science with an eye to saying, “First of all, let me ask the easy questions, the ones that don’t depend on too much. First I understand those things, and then later on we’ll get to the other ones.” And maybe never, but there’s a sort of rational order in which you, would ask questions. And so science in some sense, has to be realistic in what its goals are.

STROGATZ: Hmm, so then the resolution to our earlier question about why is science even possible, if I’m hearing you right, you’re suggesting that some things in nature could be described by the adage that you hear people say all the time, “Everything depends on everything else.” And some things in nature are not like that; not everything depends on everything else.

Am I on the right track there? That the ones where everything does depend on everything else are really going to be hard.

GOLDENFELD: Yeah. Yeah. And there’s no shortcut. And there’s other things where, if you ask the question in the right way, you can get an interesting answer, which is useful and it helps your understanding of the phenomena and so on. But if you want to know, you know, what the actual number is in degrees Fahrenheit, well, it’s not going to tell you that.

STROGATZ: Hmm. So then it seems like we’re coming to what some people might view as a disappointing cop-out of an answer, which is that science is possible because we restrict ourselves to the questions that have this kind of separation or an insulation that lets us do calculations where what’s happening here doesn’t depend on what’s happening out at Alpha Centauri.

And so it’s like we can answer the things that are easy in this sense, that they’re well separated. The others are just going to be hopeless forever? Is that the idea?

GOLDENFELD: Well, I don’t think it’s a cop-out. I think it’s a great advance to be able to say, “This question here, that’s an example of one of those things that you shouldn’t ask. And this question here is an example of one that you should.” So about 15 years or so ago, we came up with a theory that explains why there is one genetic code. It’s a general theory about the ability to express genes and make proteins and that’s what the genetic code is for, and also, by the way, it explains how life could have evolved so rapidly early on. So, it’s quite an interesting theory. And so often I’ll go and give a talk about this work, and people will ask me, “Well, why are there 20 amino acids of life?” OK? And I’ll say, “I haven’t a clue.”

[STROGATZ laughs]

GOLDENFELD: And so I think that’s an example of one of those questions that you shouldn’t ask. And I’ve got another reason for saying that. So the genetic code is literally a code book that goes from DNA — or actually, messenger RNA to amino acid, that then gets linked into a protein.

So it’s a kind of grammar, a language of molecular biology. So Francis Crick, who of course had with [James] Watson discovered the structure of DNA, wanted to try to understand why there are 20 amino acids in life. And he came up with an amazing and beautiful theory, which is mathematical. Can I tell you what the theory is? I don’t know if you know about it.

STROGATZ: So 20 amino acids, and there’s a theory for why 20?

GOLDENFELD: Yeah. So the theory is very simple. If you have a sequence of letters in threes — ACG, TAC, whatever, these correspond to certain nucleotides — you don’t know where the sequence starts. So really, whenever you read the genome, you should put commas in to tell you where the words start. So, Crick asked the question, ”Well, can you make a code so that if I’ve got four nucleotide bases, what is the largest number of amino acids you can code for so that every string in this code can make sense without you having to put in the commas?”

STROGATZ: It’s a very natural question, a beautiful question.

GOLDENFELD: It’s a beautiful question, and he came up with an answer. And the answer was that the largest number of amino acids you can get is 20. Hence, 20 amino acids of life. So then you can enumerate all of these codes without commas that Francis Crick had postulated. You can enumerate them. And when the actual genetic code was discovered by [Marshall] Nirenberg and others five or six years later, the actual code is not one of the ones that he had predicted. It’s completely wrong.

So, this is, if you like, the reasonable ineffectiveness of mathematics in biology. Because, in fact, the real code is a product of evolution. And there’s nothing special about the number 20.

So, this is an example of, you’ve got to ask the right question. You thought you could do science, biology in this case, using the same sort of elegant mathematical principles that are so powerful in physics, but you completely get egg on your face when you try them, without really understanding more about the scientific phenomena that are relevant in biology.

STROGATZ: And so would you generalize, then, to say that the role of history or evolution or contingency, those kinds of things, are another ingredient for why we might expect certain subjects to be difficult, or maybe not amenable to the elegance of math? Is that the issue?

GOLDENFELD: Well, it is, but it’s not completely hopeless. I mean, we did make a theory for the evolution of the genetic code, which did explain, you know, how is it that the world started 4.6 billion years ago? The last universal common ancestor of all life on Earth today was around 3.8 billion years ago.

So that means that in less than a billion years, life went from nothing to the architectural complexity of the modern cell. And then after that hardly evolved at all.

OK, so that is staggering. I mean, I don’t know, of course, the ultimate reason of how life evolved and so on, but at least when we made our theory of this, it explained why it evolves so rapidly, and it explained why the genetic code is so accurate and why there’s only one of them. So it explains some things, but not the other.

So we definitely understood — advanced in our understanding of basic science, but we were able to do that because we fully recognized, “Here’s a question that’s not going to be a good one to go after. Here’s a question that we might be able to do.” And I think one of the jobs of the scientist is to really ask the right questions in the right way. And that’s harder than it looks.

STROGATZ: Oh, that’s a very, very marvelous stopping point for us in a way, that part of the secret of science is the art of asking the right questions. There’s even a book with that name, isn’t there? Isn’t that Peter Medawar’s book, The Art of the Soluble?

GOLDENFELD: That’s right, but yes, I mean all science starts with asking questions. And if you don’t know how to ask questions, you can’t do science. Science is not the technology, the techniques of doing science. Of course, that’s how we’re able to do it, but fundamentally, it comes from asking questions.

STROGATZ: Probably a lot of our listeners are thinking: What about everything that’s going on today with machine learning, artificial intelligence, the possible existence of quantum computers that’s supposed to solve all kinds of problems once they really start to get serious?

Do you think that those kinds of technologies will help us deal with these intricately interwoven kinds of problems where everything or many things depend on each other, and we don’t have a good separation.

GOLDENFELD: Yes. Well, there’s two things I want to say about that. I’m really glad you raised that issue. So one of the things is the phenomenon of emergence. I mean, when people started building, you know, things like ChatGPT and so on, what those things are, are basically machines that can predict the next word. And nobody expected that those machines could pass the bar exam or medical exams or help people with their homework or help people write computer programs and so on. The range of applications has been staggering and a surprise even to the people who built these machines. And in fact, nobody really knows how they work.

In fact, if you look at the effectiveness of AI in solving problems, it also exhibits parallel relationships very much like the ones that you see in phase transitions. So one of the things I think is a great frontier for science is trying to understand how these machines are able to do so much more than what they were designed to do.

The other thing, where I think it’s important, is that what AI is very good at is discerning patterns in data, which are so complex that we don’t perceive as well as these machines.

So I think there’s great opportunity to use them to solve problems, which are very, very hard. The problem that I think is an ambitious problem I think could only be solved by using AI is trying to understand the origin of instinct. So how is it that instincts are coded in biological organisms? OK? We understand the genetic code and we understand how the proteins that go into living organisms, how they’re coded and so on.

But going from that level of description to the complexity of an organism like a fish that knows where to swim to in order to go to its breeding ground and seagulls and things like this. Clearly, we’ve somehow managed to encode very, very complex behavior. So this is something that reaches across all scales of living systems. And it’s hard for me to see, in principle, how something as complicated as instinct can be coded, but I think that AI would be able to perhaps be a tool that we could use to help us make a scientific discovery and not just, you know, build amazing technological machines.

STROGATZ: Hmm. Well, that is fascinating, Nigel. I knew it would be provocative and stimulating to talk to you and you’ve just, I think, demonstrated how, the art of science is asking good questions with that question you’ve left us with. So thank you. We’ve been speaking with physicist Nigel Goldenfeld. It has been a really great pleasure to talk to you today. Thank you.

GOLDENFELD: Thank you.

STROGATZ: Thanks for listening. If you’re enjoying “The Joy of Why” and you’re not already subscribed, hit the subscribe or follow button where you’re listening. You can also leave a review for the show — it helps people find this podcast.

“The Joy of Why” is a podcast from Quanta Magazine, an editorially independent publication supported by the Simons Foundation. Funding decisions by the Simons Foundation have no influence on the selection of topics, guests or other editorial decisions in this podcast or in Quanta Magazine.

“The Joy of Why” is produced by PRX Productions; the production team is Caitlin Faulds, Livia Brock, Genevieve Sponsler, and Merritt Jacob. The executive producer of PRX Productions is Jocelyn Gonzales. Morgan Church and Edwin Ochoa provided additional assistance. From Quanta Magazine, John Rennie and Thomas Lin provided editorial guidance, with support from Matt Carlstrom, Samuel Velasco, Nona Griffin, Arleen Santana and Madison Goldberg.

Our theme music is from APM Music. Julian Lin came up with the podcast name. The episode art is by Peter Greenwood and our logo is by Jaki King and Kristina Armitage. Special thanks to the Columbia Journalism School and Bert Odom-Reed at the Cornell Broadcast Studios.

I’m your host, Steve Strogatz. If you have any questions or comments for us, please email us at quanta@simonsfoundation.org..

[Theme fades out]