Engineering Newsletters

Mikes Notes

I recently discovered and subscribed to a series of software engineering newsletters. Most have a website and often reference other engineering websites. All are written by highly experienced engineers.

Newsletters

Just Use Postgres for Everything

By Stephan Schmidt

Just Use Postgres for Everything

How to reduce complexity and move faster

TLDR; just Postgres for everything.

We have invited complexity through the door. But it will not leave as easily.

There is Radical Simplicity though.

One way to simplify your stack and reduce the moving parts, speed up development, lower the risk and deliver more features in your startup is “Use Postgres for everything”. Postgres can replace—up to millions of users—many backend technologies, Kafka, RabbitMQ, Mongo and Redis among them.

This makes every application easier to develop, scale and operate.

Less moving parts means fewer developers for parts that don’t provide value or just replicate existing functionality (frontend) and more developers on parts (like the backend) that does provide value to customers. What if you could increase feature output by 50% without higher costs? For developers: What about a lower cognitive load? You really deeply understand all moving parts? No more imposter syndrome?

Just Use Postgres

  • Use Postgres for caching instead of Redis with UNLOGGED tables and TEXT as a JSON data type. Use stored procedures or do as I do, use ChatGPT to write them for you, to add and enforce an expiry date for the data just like in Redis.
  • Use Postgres as a message queue with SKIP LOCKED instead of Kafka (if you only need a message queue). Or as a job queue in Go with River
  • Use Postgres with Timescale as a data warehouse.
  • Use Postgres with pg_analytics as an in memory OLAP with Apache Datafusion
  • Use Postgres with JSONB to store Json documents in a database, search and index them - instead of Mongo.
  • Use Postgres as a cron demon to take actions at certain times, like sending mails, with pg_cron adding events to a message queue.
  • Use Postgres for Geospacial queries.
  • Use Postgres for Fulltext Search instead of Elastic.
  • Use Postgres to generate JSON in the database, write no server side code and directly give it to the API.
  • Use Postgres with auditing with pgaudit
  • Use Postgres with a GraphQL adapter to deliver GraphQL if needed.

There I’ve said it, just use Postgres for everything.

Resources

Good code is rarely read

From Alex Molas

Good code is rarely read

06/06/2024

The other day, I was interviewing a developer for a position at Wallapop. The candidate was a bit junior, but I enjoyed their technical assignment, and the conversation was going great. One of the questions I had to ask was

How do you define good code?

I was expecting an answer in the lines of “best practices, DRY, SOLID, code formatting, design patterns, etc.”, but the candidate just answered.

Good code is code that’s easy to read.

And this apparently simple answer got me thinking for the last week. Here are some random thoughts I had.

Why best practices?

Best practices are tools that help us write code that’s easy to read. These practices, such as the DRY principle, following SOLID, and maintaining consistent code formatting, are not arbitrary rules. These guidelines have been developed to ensure that code is understandable and maintainable. When code is easy to read, it’s easier to debug, extend, and refactor. This readability allows other developers (or even your future self) to quickly grasp the logic and purpose of the code without extensive documentation.

Read to write

But no one reads code just for the fun of it 1. When you read code, it’s to understand it and then use it. This means that the purpose of code readability is to facilitate further development. If code is written clearly, it reduces the cognitive load on the developer who will work with it. Code that is easy to read is also code that is easy to use. When functions and classes are named appropriately and their purposes are clear, you can use them without understanding their internal workings.

Code shouldn’t be read more than written

The saying “code is read more than written” is often cited to emphasize the importance of writing readable code. But it could be a symptom of bad code. If a code needs to be read frequently, it might indicate that it is not as clear or intuitive as it should be. Good code should be used more than read. It should be so well-designed that developers can use it without needing to read through it extensively.

Good code is rarely read

Therefore, your goal when writing code is for it to be read as little as possible. This might sound counterintuitive, but if your code is so easy to use that it doesn’t require an in-depth undertanding to be used, then it is good code. It should be so well-structured and named that its purpose and functionality are immediately apparent. This minimizes the need for others to read through and interpret the code, allowing them to use it more effectively.

In conclusion, good code is rarely read. Good code is so ergonomic that you almost don’t need to read it; you just use it. It allows developers to focus on building features and solving problems rather than deciphering existing code. This is the ultimate goal of good coding practices: to create code that is so clear and intuitive that it almost disappears, allowing the functionality to shine through.

CommonMark

From https://commonmark.org/

CommonMark

A strongly defined, highly compatible specification of Markdown

What is Markdown?

It’s a plain text format for writing structured documents, based on formatting conventions from email and usenet.

Who created Markdown?

It was developed in 2004 by John Gruber in collaboration with Aaron Swartz. Gruber wrote the first markdown-to-html converter in Perl, and it soon became widely used in websites. By 2014 there were dozens of implementations in many languages.

Why is CommonMark needed?

John Gruber’s canonical description of Markdown’s syntax does not specify the syntax unambiguously.

In the absence of a spec, early implementers consulted the original Markdown.pl code to resolve these ambiguities. But Markdown.pl was quite buggy, and gave manifestly bad results in many cases, so it was not a satisfactory replacement for a spec. Markdown.pl was last updated December 17th, 2004.

Because there is no unambiguous spec, implementations have diverged considerably over the last 10 years. As a result, users are often surprised to find that a document that renders one way on one system (say, a GitHub wiki) renders differently on another (say, converting to docbook using Pandoc). To make matters worse, because nothing in Markdown counts as a “syntax error,” the divergence often isn’t discovered right away.

There’s no standard test suite for Markdown; MDTest is the closest thing we have. The only way to resolve Markdown ambiguities and inconsistencies is Babelmark, which compares the output of 20+ implementations of Markdown against each other to see if a consensus emerges.

We propose a standard, unambiguous syntax specification for Markdown, along with a suite of comprehensive tests to validate Markdown implementations against this specification. We believe this is necessary, even essential, for the future of Markdown.

That’s what we call CommonMark.

Who are you today?

We’re a group of Markdown fans continually working toward the vision of CommonMark — a standard, interoperable and testable version of Markdown.

  • John MacFarlane, jgm@berkeley.edu
  • Martin Woodward, martinwoodward@github.com
  • Jeff Atwood, jatwood@codinghorror.com

Who were you in 2014, when this started?

We’re a group of Markdown fans who either work at companies with industrial scale deployments of Markdown, have written Markdown parsers, have extensive experience supporting Markdown with end users – or all of the above.

  • John MacFarlane, of Pandoc
  • David Greenspan, of Meteor
  • Vicent Marti, of GitHub
  • Neil Williams, of Reddit
  • Benjamin Dumke-von der Ehe, of Stack Overflow / Stack Exchange
  • Jeff Atwood, of Discourse

How can I help?

Exercise our reference implementations, or find a community implementation in your preferred environment or language. Provide feedback!

If a CommonMark implementation does not already exist in your preferred environment or language, try implementing your own CommonMark parser. One of our major goals is to strongly specify Markdown, and to eliminate the many old inconsistencies and ambiguities that made using Markdown so difficult. Did we succeed?

Where can I find it?

spec.commonmark.org

The CommonMark specification.

code.commonmark.org

Reference implementation and validation test suite on GitHub.

talk.commonmark.org

Public discussion area and mailing list via Discourse.

commonmark.org/help

Quick reference card and interactive tutorial for learning Markdown.

spec.commonmark.org/dingus/

Live testing tool powered by the reference implementation.

When is the spec final?

The current version of the CommonMark spec is quite robust after many years of public feedback.

There are currently CommonMark implementations for dozens of programming languages, and the following sites and projects have adopted CommonMark:

  • Discourse
  • GitHub
  • GitLab
  • Reddit
  • Qt
  • Stack Overflow / Stack Exchange
  • Swift

Payment Service Providers

Mikes Notes

Eventually, Ajabbi will need the services of a payment service provider.

"A payment service provider (PSP) is a third-party company that allows businesses to accept electronic payments, such as credit card and debit card payments. PSPs act as intermediaries between those who make payments, i.e. consumers, and those who accept them, i.e. retailers."

"They often provide merchant services and act as a payment gateway or payment processor for e-commerce and brick-and-mortar businesses. They may also offer risk management services for card and bank-based payments, transaction payment matching, digital wallets, reporting, fund remittance, currency exchange and fraud protection. The PSP typically provides software to integrate with e-commerce websites or point of sale systems." - Wikipedia.

Resources

Using CloudConvert

Mikes Notes

CloudConvert is an excellent file conversion SaaS service. This is the best service available, its fast, and I use the free version most days. Free means less than 25 conversions per day. I will eventually get the paid version. There is an API.

Description

"CloudConvert is an online file converter. We support nearly all audio, video, document, ebook, archive, image, spreadsheet, and presentation formats. To get started, use the button below and select files to convert from your computer." - CloudConvert

"CloudConvert was founded 2012 with the vision to build a universal tool for file conversions. Our product is used both by end users and corporate customers via our API. CloudConvert is built by Lunaweb GmbH, a company located in Munich, Germany.

We believe that it is important to focus on core business competencies and let others do everything else. This is why we are doing what we are best at doing: converting files.

We believe that data security is the most important good in our industry. This is why data protection and security have the highest priorities for us.

We believe in small, agile and unbureaucratic teams."

Pricing

"You can use CloudConvert absolutely free for up to 25 conversions per day. Beyond that we offer flexible payment options. Use the quantity slider below to customize a package or subscription according to your needs." - CloudConvert

Universally Unique Identifier (UUID)

"A Universally Unique Identifier (UUID) is a 128-bit label used for information in computer systems. The term Globally Unique Identifier (GUID) is also used, mostly in Microsoft systems. [1][2]

When generated according to the standard methods, UUIDs are, for practical purposes, unique. Their uniqueness does not depend on a central registration authority or coordination between the parties generating them, unlike most other numbering schemes. While the probability that a UUID will be duplicated is not zero, it is generally considered close enough to zero to be negligible.[3][4]

Thus, anyone can create a UUID and use it to identify something with near certainty that the identifier does not duplicatAe one that has already been, or will be, created to identify something else. Information labeled with UUIDs by independent parties can therefore be later combined into a single database or transmitted on the same channel, with a negligible probability of duplication.

Adoption of UUIDs is widespread, with many computing platforms providing support for generating them and for parsing their textual representation." - Wikipedia

Versions

The OSF DCE variant defines eight "versions" in the standard, and each version may be more appropriate than the others in specific use cases.

  • Versions 1 and 6 (date-time and MAC address)
  • Version 2 (date-time and MAC address, DCE security version)
  • Versions 3 and 5 (namespace name-based)
  • Version 4 (random)
  • Version 7 (timestamp, counter and random)
  • Version 8 (custom)

Collisions

For example, the number of random version-4 UUIDs which need to be generated in order to have a 50% probability of at least one collision is 2.71 quintillion, computed as follows:



This number is equivalent to generating 1 billion UUIDs per second for about 86 years. A file containing this many UUIDs, at 16 bytes per  UUID, would be about 45 exabytes.

The smallest number of version-4 UUIDs which must be generated for the probability of finding a collision to be p is approximated by the formula

Thus, the probability to find a duplicate within 103 trillion version-4 UUIDs is one in a billion.

Issues

Collisions have occurred when manufacturers assign a default UUID to a product, such as a motherboard, and then fail to over-write the default UUID later in the manufacturing process. For example, UUID 03000200-0400-0500-0006-000700080009 occurs on many different units of Gigabyte-branded motherboards.

Common Design Patterns at Stripe

Mikes Notes

An excellent series of articles on designing APIs from Stripe, The Payment Service Provider. There are detailed examples and coding conventions.

Designing APIs for Humans Series' Articles by Paul Asjes (Stripe Dev)

Examples

$pi = $stripe->paymentIntents->create([
  'amount' => 1000,
  'currency' => 'usd',
  // Without prefixes, we'd have to supply a 'type'
  'payment_method' => [
    'type' => 'card',
    'id' => '1LaRQ7GUcADgqoEMV11wEUxU'
  ],
]);

Docs

Stripe also has superb Developer Docs, which are a helpful format to start with when designing Pipi Docs, especially for the API.

Resources

Other stuff

Dashboard Design Patterns

Mikes Notes

Vitaly Friedman, editor of Smashing Magazine, recently posted resources about Dashboard Design Patterns.

"This page lists design patterns for dashboard design collected to support the design and creative exploration of dashboard design. We run a dedicated workshop in March 2022 to help you applying and discussing design patterns in your work.

The patterns are coming out of our research, described in our publication:

Bach, Euan Freeman, Alfie Abdul-Rahman, Cagatay Turkay, Saiful Khan, Yulei Fan, Min Chen: Dashboard Design Patterns, IEEE VIS Conference / Journal of Transactions on Visualization and Computer Graphics (TVCG), 2023. .

What are Dashboards?

Dashboards offer a curated lens through which people view large and complex data sets at a glance. They combine visual representations and other graphical embellishments to provide layers of abstraction and simplification for numerous related data points, so that dashboard viewers get an overview of the most important or relevant information, in a time efficient way. Their ability to provide insight at a glance has led to dashboards being widely used across many application domains, such as business, nursing and hospitals, public health, learning analytics, urban analytics, personal analytics, energy and more." - https://dashboarddesignpatterns.github.io/


Resources

Theorem Provers

Mikes Notes

I have a fortnightly online meeting with my computer scientist friends Alex and Chris to discuss our various computer and mathematics projects. It is an excellent way of getting feedback and being exposed to new ideas.

Earlier this year, Alex mentioned an online talk at the London Mathematical Society that he attended. I then watched it. It was about "proving assistants," AKA "theorem provers."

I find mathematics papers unfathomable. But I could follow along quite easily reading the proofs generated by Isabella, the proving assistant.

"Automated theorem proving (also known as ATP or automated deduction) is a subfield of automated reasoning and mathematical logic dealing with proving mathematical theorems by computer programs. Automated reasoning over mathematical proof was a major impetus for the development of computer science." - Wikipedia.

Isabella

"Abstract: The formalisation of mathematics is an ongoing process that arguably started as early as the 19th century, intensified with the foundational crisis at the start of the 20th century, and since the 1970s has been conducted with the help of computers. Recent decades have seen the machine formalisation of lengthy and technically complicated proofs, but some have argued that even these were not representative of modern mathematics. Recent achievements by a number of different groups are starting to challenge this scepticism. The speaker will outline some of these, while also noting some of the remaining trouble spots.

This was the LMS-BCS FACS Seminar, which took place on Monday 15 January 2024 online via Zoom." - LMS YouTube Channel

Vampire

"Abstract: Vampire is a fully automated theorem prover for first-order logic that has been developed for over 25 years. It has a long tradition of being the “best” in the world (solves the most in the least time at the theorem prover competition) for traditional first-order logic problems found in many domains. However, this traditional first-order logic setting often struggles to capture problems from program verification that need to talk about theories such as arithmetic. Indeed, program verification is typically the domain of SMT solvers (Boolean SAT solvers extended with theory decision procedures). However, we often see problems in program verification that combine theories and quantification (the bread-and-butter of traditional first-order logic reasoning) where SMT solvers struggle with heuristic approaches. Therefore, around 10 years ago we embarked on a journey to extend Vampire to the program verification space, adding theory reasoning, induction, interpolation, and a some other fun things. It is this journey, the challenges, advances, and applications, that I will talk about in this presentation

Part of the LMS Computer Science Colloquium 2023, which took place on Friday 1 December 2023 at De Morgan House, London and online via Zoom." - LMS YouTube Channel

Resources

Example in Isabella

For example, a declarative proof by contradiction in Isar that the square root of two is not rational can be written as follows.
theorem sqrt2_not_rational:
  "sqrt 2 ∉ ℚ"
proof
  let ?x = "sqrt 2"
  assume "?x ∈ ℚ"
  then obtain m n :: nat where
    sqrt_rat: "¦?x¦ = m / n" and lowest_terms: "coprime m n"
    by (rule Rats_abs_nat_div_natE)
  hence "m^2 = ?x^2 * n^2" by (auto simp add: power2_eq_square)
  hence eq: "m^2 = 2 * n^2" using of_nat_eq_iff power2_eq_square by fastforce
  hence "2 dvd m^2" by simp
  hence "2 dvd m" by simp
  have "2 dvd n" proof -
    from ‹2 dvd m› obtain k where "m = 2 * k" ..
    with eq have "2 * n^2 = 2^2 * k^2" by simp
    hence "2 dvd n^2" by simp
    thus "2 dvd n" by simp
  qed
  with ‹2 dvd m› have "2 dvd gcd m n" by (rule gcd_greatest)
  with lowest_terms have "2 dvd 1" by simp
  thus False using odd_one by blast
qed

The Stanford Encyclopedia of Philosophy

Mikes Notes

The Stanford Encyclopedia of Philosophy (SEP) might be a helpful resource. Some science papers I have to read to build Pipi are very academic. 

"Welcome to the Stanford Encyclopedia of Philosophy (SEP), which as of Summer 2023, has nearly 1800 entries online. From its inception, the SEP was designed so that each entry is maintained and kept up-to-date by an expert or group of experts in the field. All entries and substantive updates are refereed by the members of a distinguished Editorial Board before they are made public. Consequently, our dynamic reference work maintains academic standards while evolving and adapting in response to new research. You can cite fixed editions that are created on a quarterly basis and stored in our Archives (every entry contains a link to its complete archival history, identifying the fixed edition the reader should cite). The Table of Contents lists entries that are published or assigned. The Projected Table of Contents also lists entries currently unassigned but nevertheless projected." - SEP

Resources

How Signiant documents their API

Mikes Notes

I found this API documentation page. It is generated by Swagger, and it looks great.

"Signiant Flight Deck provides a REST API to automate user management, user group, agent groups, jobs, resource controls, and job groups through the Manager software." - Signiant

List of API Endpoints


API Endpoint Example


The code

[
  {
    "user": {
      "userName": "exampleuser",
      "fields": {
        "accessAcls": [
          {
            "edit": true,
            "execute": true,
            "read": false,
            "remove": true,
            "userName": "admin"
          }
        ],
        "cell": "1-555-555-1234",
        "email": "user@example.com",
        "failedLoginTimePeriod": "24",
        "fax": "1-555-555-4321",
        "firstName": "Example",
        "groups": "exampleGroup",
        "ignoreAdministrationForSoapAuth": false,
        "lastName": "User",
        "maxAllowedLoginAttempts": "10",
        "menus": [
          {
            "name": "Groups",
            "parent": {
              "name": "Jobs"
            }
          }
        ],
        "organization": "Example Organization",
        "password": "aValidPassword1",
        "phone": "1-555-555-1111",
        "roles": {
          "isTemporary": false,
          "isAdminGuiLogin": false,
          "validFrom": null,
          "validTo": null,
          "isSysAdmin": false,
          "isOrgAdmin": false,
          "isWorkflowAdmin": false,
          "isWorkflowSupervisor": false,
          "isCompEditor": true,
          "isMonitorUser": false
        },
        "title": "Dr."
      }
    }
  }
]

Mikes Notes

For security, use both

  • API key
  • Access tokens from an authorisation server

Web app manifests

Mikes Notes

I recently used Favicon.io, an online app to create the favicons for the Ajabbi websites. One of the generated files was called site.webmanifest, which I had never heard of before. Everything below comes from MDN.

This is not in production.

Notes

"A web application manifest, defined in the Web Application Manifest specification, is a JSON text file that provides information about a web application.

The most common use for a web application manifest is to provide information, such as the app's name and icon, that the browser needs to install a progressive web app (PWA) on a device.

A web application manifest contains a single JSON object with the top-level keys called members." - MDN.

Example

JSON
{
  "name": "HackerWeb",
  "short_name": "HackerWeb",
  "start_url": ".",
  "display": "standalone",
  "background_color": "#fff",
  "description": "A readable Hacker News app.",
  "icons": [
    {
      "src": "images/touch/homescreen48.png",
      "sizes": "48x48",
      "type": "image/png"
    },
    {
      "src": "images/touch/homescreen72.png",
      "sizes": "72x72",
      "type": "image/png"
    },
    {
      "src": "images/touch/homescreen96.png",
      "sizes": "96x96",
      "type": "image/png"
    },
    {
      "src": "images/touch/homescreen144.png",
      "sizes": "144x144",
      "type": "image/png"
    },
    {
      "src": "images/touch/homescreen168.png",
      "sizes": "168x168",
      "type": "image/png"
    },
    {
      "src": "images/touch/homescreen192.png",
      "sizes": "192x192",
      "type": "image/png"
    }
  ],
  "related_applications": [
    {
      "platform": "play",
      "url": "https://play.google.com/store/apps/details?id=cheeaun.hackerweb"
    }
  ]
}

Deployment of Manifest

Web app manifests are deployed in your HTML pages using a <link> element in the <head> of a document:

HTML

  <link rel="manifest" href="manifest.json" />


The .webmanifest extension is specified in the Media type registration section of the specification (the response of the manifest file should return Content-Type: application/manifest+json). Browsers generally support manifests with other appropriate extensions like .json

Content-Type: application/json

If the manifest requires credentials to fetch, the cross-origin attribute must be set to use credentials, even if the manifest file is in the exact origin as the current page.

HTML

<link rel="manifest" href="/app.webmanifest" crossorigin="use-credentials" />

Resources