htmx 4.0.0 has been released!

Mike's Notes

More details from Carson about htmx 4.0.

Resources

References

  • Hypermedia Controls: From Feral to Formal

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

26/09/2026

htmx 4.0.0 has been released!

By: Carson Gross
HTMX: 28/08/2026

Carson Gross: I am a programmer comfortable with both front-end & back-end development. I currently teach at Montana State University.

My primary technical interests are hypermedia and programming languages, but I’m comfortable working in most areas of enterprise systems.

...

Release announcement, 2026-08-28.

htmx 4.0.0 Release

The htmx team is very happy to announce the release of htmx 4.0.0! This is the culmination of 8 months of work (plus a game) and we are very happy with the results.

The idea of htmx 4 started to germinate when I decided to create fixi and, in doing so, got more familiar with the fetch() API and async programming in JavaScript. (htmx had always used XMLHttpRequest due to backwards compatibility issues.)

One chance evening I was contacted by Christian, who had some interesting ideas around streaming HTML that got me thinking that moving the internals to fetch() would simplify things for him and for the library in general.

After a bit of work I managed to get Michael and Alex on board, and we were off to the races.

Development has been very smooth. We started a port of fixi + the htmx test suite. Over time, we rediscovered why htmx did many of the things that it did and moved our new implementation closer and closer to the old one. At this point the behavioral differences between 2.x and 4.x are relatively small and where they do diverge we have made explicit choices that we feel will put htmx-based applications in a good spot for being 100-year web services

Note that we are not marking 4.0 as latest in NPM because we do not want to force-upgrade users who are relying on non-versioned CDN URLs for htmx. Instead, 2.x will remain latest and the 4.0 line will remain next until some point in early 2027. The website, however, will reference 4.0.

Major Changes

As mentioned above, htmx 4, from a user’s viewpoint, is almost identical to htmx 2. There are three major changes:

  • Attribute inheritance is now explicit by default rather than implicit by default (this is the biggest upgrade item)
  • The htmx event names have been standardized & cleaned up. Some advanced users may need to change the events they listen for.
  • History support now does not use localStorage by default (which was a cause of many support headaches). Most people won’t notice this at all.

Internally, we migrated from XMLHttpRequest to fetch() but that should be transparent for most users of htmx.

Attribute Inheritance

In htmx 2 many attributes were “inherited” by default. This allows you to place attributes on parent elements and their behavior will apply to child elements. This behavior, which came from the intercooler.js days, was inspired by CSS and, unsurprisingly, worked out about the same as CSS: powerful but difficult to understand at times.

In htmx 4 attributes are not inherited unless you explicitly say so by adding an :inherited after the attribute name:

<!-- htmx 2 -->
<div hx-confirm="Are you sure?">
    <button hx-delete="/item/1">Delete</button>
</div>
<!-- htmx 4 -->
<div hx-confirm:inherited="Are you sure?">
    <button hx-delete="/item/1">Delete</button>
</div>

This will be the largest upgrade burden in migrating from htmx 2 to htmx 4. To make things easier, we have provided a command line tool to find places you need to mark as inherited.

Note that attributes like hx-disinherit, etc. are no longer required and should be removed.

Events

The events triggered by htmx 2 had grown organically over the life of the library and were not particularly well organized, making it difficult to know exactly which event was fired when.

In htmx 4, all events now follow htmx:phase:action[:sub-action]:

htmx 2 htmx 4
htmx:beforeRequest htmx:before:request
htmx:afterRequest htmx:after:request
htmx:beforeSwap htmx:before:swap
htmx:afterSwap htmx:after:swap
htmx:configRequest htmx:config:request

In addition, the following changes were made:

  • Most error events collapse into htmx:error. HTTP error responses fire htmx:response:error.
  • The htmx:xhr:* events are removed. htmx 4 uses fetch().
  • The htmx:validation:* events are removed in favor of native browser form validation.

The full table is in What’s New in htmx 4.

The command line upgrade checker flags old event names in hx-on attributes and in your JavaScript where it can find them.

History

History support has always been included in htmx, allowing you to implement back-button aware actions with simple attributes. In htmx 2, a cache in localStorage was used to snapshot pages for restoration. Unfortunately a large source of issues was that this snapshot could include DOM mutations by 3rd party JavaScript libraries. When the page was restored, those mutations remained but the underlying JavaScript logic was not.

htmx 4 does not cache pages in localStorage. On back navigation htmx re-fetches the page and swaps it into <body>, or into the [hx-history-elt] element if one is present. This allows 3rd party JavaScript libraries to “just work” in most cases and, with good request caching, is very fast.

If you want local caching instead, we now ship a very complete hx-history-cache extension that restores history from sessionStorage and is designed to integrate well with scripting solutions like Alpine.js, etc.

New Features

There are two big new features in htmx 4, both of which we are really excited about:

Morph Swaps

We now support morphing swaps out of the box with htmx. I created idiomorph and nearly included it in htmx 2.x but decided against it. In htmx 4, Michael has done great work improving on that algorithm and integrating it seamlessly into htmx.

<hx-partial>

Another major new feature is the <hx-partial> tag. This tag is similar to out-of-band swaps, but is much clearer when you want to do something beyond just replacing a single element with a new version of itself:

<hx-partial hx-target="#messages" hx-swap="beforeend">
    <div>New message</div>
</hx-partial>
<hx-partial hx-target="#count">
    <span>5</span>
</hx-partial>

Extensions

Much of the excitement in htmx 4 is in the extensions. Switching to fetch() internally let us rethink how extensions can and should work, and sparked the creation (and recreation) of many new extensions, for example:

  • hx-preload - preload content (e.g. on mouseover) to speed requests up
  • hx-download - native, fetch-based file downloads
  • hx-alpine-compat - smooths over compatibility issues between htmx and Alpine.js
  • hx-history-cache - caches history in sessionStorage, provides Alpine.js compatibility

Additionally, there are three new or updated streaming HTML extensions:

  • hx-sse streams over text/event-stream.
  • hx-ws streams and sends over WebSockets.
  • hx-multipart streams over multipart/mixed

Finally, we decided it was time to try our hand at our own small front-end scripting solution that tightly integrates with htmx. hx-live is inspired by Alpine.js, jQuery and hyperscript, and makes front end scripting pleasant and fun. It even supports what we are calling DOM-based, HATEOAS-friendly reactivity.

There is a new htmax.js bundle in the distribution which packages htmx with the most popular of these in a single file if you don’t want to think about which ones you want to pick.

Upgrading

For a complete upgrade guide see What’s New in htmx 4.

As mentioned earlier, we are providing an upgrade tool to help you:

$ npx htmx.org@4.0.0 upgrade-check -- ./templates

File extensions: .html, .php, .js, .ts, .jinja, .jinja2, .j2, .erb, .hbs
Use --ext to add more (e.g. --ext .vue --ext .svelte)

Scanning 1 file(s)...

Found 8 issue(s) in 1 of 1 file(s).

  • templates/index.html:1: [inheritance] hx-headers needs :inherited suffix (descendant on line 3 has hx-delete) (this looks like a CSRF token; without :inherited the header does not reach child elements and the server rejects the request)
  • templates/index.html:2: [inheritance] hx-target needs :inherited suffix (descendant on line 3 has hx-delete)
  • templates/index.html:2: [inheritance] hx-confirm needs :inherited suffix (descendant on line 3 has hx-delete)
  • templates/index.html:3: [renamed-attr] hx-disable -> rename to hx-ignore (hx-disable now means 'disable during request')
  • templates/index.html:4: [removed-attr] hx-vars is removed -> use hx-vals with js: prefix
  • templates/index.html:4: [removed-attr] hx-prompt is removed -> load the hx-prompt extension to keep the same syntax
  • templates/index.html:9: [old-event] old event name "htmx:afterRequest" -> "htmx:after:request"
  • templates/index.html:9: [old-api] htmx.addClass() is removed -> use element.classList.add()

We are also shipping an agent skill to assist in upgrading

Installing

htmx 4.0 can be installed via a package manager referencing version 4.0.0, or can be linked via a CDN:

<script src="https://unpkg.com/htmx.org@4.0.0/dist/htmx.min.js"></script>

or Downloaded

LLMs

Like it or not, a lot of people are using LLMs and we are providing the following skills files for LLMs:

  • htmx-guidance - core htmx skills for developing with htmx 4
  • htmx-debugging - diagnosing htmx issues during dev
  • htmx-extension-authoring - writing and debugging htmx 4 extensions
  • htmx-upgrade-from-htmx2 - migrating a codebase from htmx 2.x to 4.x

(Let’s leave aside if releasing a new version of a library in the LLM era is a good or bad thing!)

Conclusion

We hope you enjoy htmx 4. htmx 2 will continue to be supported indefinitely so don’t feel any pressure to upgrade.

I’d like to thank the following people for all their help with this release:

  • Michael West - Incredible teammate & grug-brained developer
  • Christian Tanul - Inspired htmx 4 & led the streaming & live extensions
  • Alex Petros - For keeping the ship on an even keel
  • Stephen Mitchell - The genius behind the game
  • Stu Kennedy - Our WebSockets expert
  • André Ahlert Jr. - Providing IDE & Editor Support
  • Dien Hoa Truong - For kicking the tires on early htmx 4 and helping fix many bugs

Upgrade Music

Wouldn’t be an htmx update without upgrade music:

By , No comments:
Tags

htmx 4.0: a Fetch-Based Rewrite, Built-In Morphing Swaps, and Explicit Attribute Inheritance

Mike's Notes

Excellent news.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library > Subscriptions > InfoQ
  • Home > Handbook > 

Last Updated

25/09/2026

htmx 4.0: a Fetch-Based Rewrite, Built-In Morphing Swaps, and Explicit Attribute Inheritance

By: Daniel Curtis
InfoQ: 18/09/2026

Daniel Curtis: Daniel Curtis is a UI Development Manager at Griffiths Waite, a software consultancy based in Birmingham, UK. He leads front-end engineering efforts with a strong focus on delivering innovative enterprise solutions using TypeScript across the stack. Daniel is passionate about modern web architecture, developer experience, and the use of AI to both support software delivery and solve real customer problems within products.

...

The htmx team has released htmx 4.0.0, the first major version of the hypermedia library since 2.0 arrived in 2024, landing after eight months of work and skipping version 3 entirely. Creator Carson Gross, who had promised there would never be an htmx 3, summed up the leap to InfoWorld in a one word quote: "Oops."

htmx has swapped its long serving XMLHttpRequest transport for the modern fetch() API, a rewrite that unlocks native streaming while keeping the script around 14KB. For most developers the familiar hx-get and hx-post attributes behave exactly as before, but the move enables the two features the team is most excited about, which is built in morphing swaps powered by idiomorph, which preserve DOM state such as input focus during updates, and a new hx-partial tag that lets one response cleanly update several targets, a tidier take on out of band swaps.

Attribute inheritance is now explicit, so parent attributes reach children only when marked with an :inherited suffix:

<div hx-confirm:inherited="Are you sure?">
 <button hx-delete="/item/1">Delete</button>
</div>

A detailed DEV Community write up warns this is the trap to watch, because an hx-headers attribute passing a CSRF token to child elements silently stops working without :inherited and the server begins rejecting requests with a 403 while nothing looks broken.

Event names have also been standardised to a htmx:phase:action pattern, so htmx:afterRequest becomes htmx:after:request, and history no longer snapshots the DOM into localStorage, instead refetching pages so third party scripts just work.

A Hacker News thread passed 700 points, with developers praising the Go, SQLite and htmx stack and how well AI assistants handle it, one noting that Claude "understands it and does it well", and the r/htmx release post drew over 300 upvotes.

Elsewhere a .NET and Angular developer argued htmx forces you to mix presentation with business logic and another agreed and said their team is moving back to React.

htmx 4.0 ships under npm's next tag rather than latest, so nobody on an unversioned CDN link is force upgraded, and 2.x stays supported indefinitely. The team provides a command line checker that flags inheritance issues, renamed attributes and old events:

npx htmx.org@4.0.0 upgrade-check -- ./templates

Beyond inheritance changes, there is also a new 60 second request timeout and a number of attribute renames, such as hx-disable becoming hx-ignore. These changes need applying with care. A full What's New in htmx 4 guide is available, and dedicated skills files for LLM assisted upgrades.

htmx is an open source library that adds AJAX, CSS transitions, WebSockets and server sent events directly to HTML through attributes, letting developers build modern interfaces with minimal JavaScript. It remains a flagship of the hypermedia movement, popular with Django, Go and Rails backends where server rendered HTML meets low client side complexity.

By , No comments:
Tags

Accordion Icons: Which Signifiers Work Best?

Mike's Notes

Excellent NN Group article on Accordion Icons: What works and what doesn't.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library > Subscriptions > NN/g Newsletter
  • Home > Handbook > 

Last Updated

24/09/2026

Accordion Icons: Which Signifiers Work Best?

By: Page Laubheimer and Raluca Budiu
NN/g Newsletter: 23/08/2020

Page Laubheimer: Page Laubheimer is a former NN/G employee who worked on research, teaching, leadership, and design thought leadership. His expertise focused on web applications, AI, and projects in complex domains. His many research findings and recommendations were also informed by his background in library and information science, and his work often involved information architecture, navigation design, taxonomy construction, and ontology management.

Raluca Budiu: Raluca Budiu is Senior Director, Data Strategy, at Nielsen Norman Group, where she uses her data-analysis expertise to drive strategic decisions. She also serves as editor for the articles published on NNgroup.com. Raluca has coauthored many NN/g reports, as well as the book Mobile Usability. She holds a Ph.D. from Carnegie Mellon University.

...

Summary:

The caret icon most clearly indicated to users that it would open an accordion in place, rather than linking directly to a new page.

For good reasons, accordions are a popular UI element today: on mobile, they are an essential tool because they collapse content and make page length manageable, but even on desktop, they mitigate visual complexity and allow users to focus on the content most relevant for the task at hand (and are particularly appropriate in complex applications).

A question that is often asked in both our Mobile UX and Application Design classes is: which icon should we use to best signal that content will expand in place? In other words, what’s the best signifier for accordions?

We decided to investigate this question as part of a bigger study of navigation and subnavigation on mobile. We looked at several possible icons as signifiers for accordions:

  • Caret (or downward-facing arrow)
  • Plus
  • Right-facing arrow
  • No icon at all

There are some subtle differences in what these icons are commonly used to represent — while the caret and plus icons are typically meant to indicate that an accordion will open, designers have used the right-facing arrow icon to signal two different actions: either staying on the same page and expanding content, or visiting a different page. Also, after expansion, the caret will typically twist (maybe in a nice quick animation), and a plus sign will turn into a minus, to become a signifier for the reverse action of collapsing the newly expanded content.

Jennair: Accordion signaled by caret icon

Bolé Road Textiles: accordion signaled by plus icon

Ferrari: accordion signaled by arrow icon

Braun: accordion with no signifier icon

We performed a quantitative study to find out which of these icons is the most effective at signaling that it will open an accordion.

In This Article:

  • Methodology
  • Where Participants Tap
  • What Participants Expect to Happen
  • Key Findings
  • Design Recommendations
  • Summary

Methodology

Materials. We decided to study the accordions in the context of mobile navigation so we created 11 prototypes of mobile websites, each in a different type of industry — clothing ecommerce, big-box retail, auto parts, finance, news, local government, higher education, DIY home projects, consumer-goods reviews, healthcare, and travel. The prototypes were not interactive; they were simple mockups with the menu already open and a list of categories visible on the screen, to ensure that we were only measuring the interaction with the accordions, as opposed to users’ efforts to locate the menus, look at homepage content, and so forth.

For each prototype, we created a task that involved finding information in one of the accordions visible in the prototype. For example, on a prototype of a consumer-goods review website, the task was “Find reviews of dishwashers.”

We also created 5 different variations for each prototype; each variation used one of 4 possible icons (an arrow, a caret, a plus, a foil) or no icon. We created a foil icon (that had not been used before for accordions) in order to see whether the type of icon mattered at all or the presence of an icon next to the accordion name was enough to signal an accordion (but the icon itself did not matter).

The icons tested in this study included a downward-facing caret, a right-facing arrow, a plus, a nonsense foil icon (used as a control for comparison), or a blank space. The icons were randomly assigned for each task, meaning that participants saw a random assortment and order of these five icons.

An example of one of the prototypes, shown in all 5 icon variations

We placed the accordion icon to the right of its associated label, close to the edge of the screen and right aligned. (We acknowledge that another possible position is to the left of the label — our results may not generalize to that placement.)

As the length of the text labels might impact where users look or click, we ensured that label lengths in each prototype were distributed equally and that the “correct” answer for each task (i.e. the menu option that was likely to be chosen by users) was of a different length in each prototype. 

We also ran several pilot studies to ensure that the task success rate was high (over 90%) and that people would be able to easily identify the “correct” menu category that had to be selected in order to complete each task. After each pilot study we refined the task phrasing and labels in the menu. Thus, effectively, we ran several usability studies on our prototypes’ information architecture before we collected the quantitative data. This process ensured that figuring out the right category was relatively trivial for most participants — as our study goal was to test the accordion signifiers, not the difficulty of finding the information in the tasks we gave users.

Participants. Our study had 136 participants.  All participants saw instances of all the different icons (in different prototypes, and in different order).

Procedure. The test was conducted on UserZoom as a series of first-click tasks, administered on mobile devices. All participants were shown one version of each of the 11 prototypes and the associated task in a random order. After the participants indicated where they would tap to find the answer, they had to answer a multiple-choice question asking what they expected to happen (going directly to a new page, seeing additional menu options on the same page, or something else). 

We collected data on:

  • Tap location: Where the participant tapped (directly on the text label, directly on the icon, in the space in-between the label and the icon, or anywhere else on the prototype)
  • Participants’ expectations: The response to the post-task question about expectations (going directly to a new page, seeing an additional list of links on the same page, or something else)

Where Participants Tap

Taps outside the area associated with the accordion were relatively few (5–8%), and most taps fell either on the label or on the icon associated with the accordions (and not on the space in between the two).

Participants were equally likely to tap on either the text label or the icon, except for the caret icon, where there was a statistically significant tendency to tap on the icon over the text label. When no icon was present, users were much more likely to tap on the text label than the empty space where an icon would normally be.

An ANOVA on the data for when users tapped on the text label found a significant effect of icon type both when participants or prototypes were treated as the random factor. Pairwise contrasts indicated that the no-icon condition was significantly different than all other conditions.  In particular, there was a higher chance to tap on the label when no icon was present — a finding that is not surprising at all since the “icon area” was just empty space in this condition (we just tracked if people tapped on the place where an icon would normally be), but an important reminder that users typically choose to interact with obvious signifiers.

For none of the icon conditions was there a statistically significant tendency to tap on the icon more than on the label except for the caret (p<0.01) —  in other words, when the accordion was signaled by a caret, people tended to tap more on the icon than on the label. For all other icons (arrow, foil, plus), there was no statistically significant preference for the icon.

What Participants Expect to Happen

To analyze the responses to the post-task question (regarding expectations to stay on the page) we defined the new-page expectation as a binary variable quantifying  whether participants expected to stay on the same page (0) or go to a new page (1). A rate of over 50% for a particular signifier indicates that overall people expected to go to a new page. For an accordion, where we want to convey that the page won’t change, the rate should ideally be under 50%.

None of the standard-icon rates were significantly different from 50%, indicating that there was no strong expectation related to them — people did not necessarily expect to stay on the page or leave the page. With the foil and the no-icon conditions, however, there was a significant expectation that people will leave the page.

For the standard signifiers (caret, plus, and arrow), there was no strong expectation to leave the page (as the rate of new-page expectation is not significantly different than 50%, p >0.05).  Among all the standard signifiers, the caret has a significantly stronger expectation to stay on the page than the foil (p<0.05) or the no-icon condition (p <0.05) and the plus is significantly better than the foil (p<0.05), but not than the no-icon condition. The arrow is not statistically different than the foil or no-icon conditions, suggesting that this icon should not be used for accordions.

Key Findings

Our hypothesis was that having no icon would cause users to expect that tapping on the menu item would take them directly to a new page, and this hypothesis held up in our study. Likewise, the foil icon was presumed to not have any association for users with opening an accordion, and this also held up.

Then, we compared users’ reported expectations of the caret, plus, and arrow icons to the foil or no icon, as a test of whether or not they were superior signifiers for an accordion. We also looked at whether people tend to tap the label or the icon for these different signifiers, under the assumption that, if indeed there is a strong tendency to tap only on one of them, we could, perhaps, separate the functionality of the two (a la split buttons).

  • For accordion patterns, people tend to tap fairly equally on the icon and on the label for most standard icons. The only exception is the caret, where people tend to tap somewhat more on the icon, but still 29% of the taps fall on the label. This suggests it’s not safe to use split buttons for accordions — where the text label links directly to a landing page, but the icon opens the accordion.
  • Making up new icons for accordion signifiers or using no signifiers at all is not a good idea as they violate users’ expectations (which are that they will be taken to a new page).
  • None of the standard signifiers have a strong association with staying on the page. That being said:
    • Using a caret is definitely better than using no icon at all or a random icon in terms of conveying the expectation to stay on the page (and open an accordion).
    • Using an arrow or a plus is not better than using no signifier at all.

  • When no icon was present, users tended to tap on the text label, rather than the empty space in that row. While an obvious finding, this is more evidence that users tend to interact with strong, clear signifiers.
  • Interestingly, using a right-facing arrow icon (as opposed to a plus or caret) was NOT significantly associated with an expectation of going directly to a new page.  While many designers might think that an arrow icon implies “go directly to a page” whereas the same icon pointing down implies “open an accordion here on the page” that was not supported in our data.

Design Recommendations

  • If using accordions in your mobile menus, the caret appears to be the safest icon choice.
  • Do not have icons and text labels link to different actions (i.e. label directly to page, icon opening an accordion).  Our study findings further support our previous recommendations that you should not use split buttons for accordions. Our study showed that users tapped fairly equally on both the text label and the icon and did not expect them to do different things.
  • Decide if your menu items will either open a submenu accordion or go directly to a category-overview page.  If you choose to have your menu items link directly to a landing page, do not use a right-aligned icon.

Summary

Users tend to click fairly equally on both the accordion icon and the accordion label, so avoid dissociating those by assigning them different functionalities.  Use a caret icon to designate an accordion, whether on desktop or mobile — our study found that of the standard set of icons used in this context, only the caret performed better than either no icon or a nonsense icon at indicating that than it was an accordion.

By , No comments:
Tags ,

Ajabbi mission sign-up

Mike's Notes

While visiting Christchurch recently, I met with people who would like to help with Ajabbi's mission in the future. People in other countries have also reached out.

If this is you, please use this Google Form as a simple first step. Then we'll contact you individually by email to set up a follow-up chat in person, by phone, or by video call in English.

If you want to get free early access to Pipi to help with testing and give feedback, go here.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > Ajabbi > Mission

Last Updated

23/09/2026

Ajabbi mission sign-up

By: Mike Peters
On a Sandy Beach: 22/09/2026

Mike invented and designed Pipi and founded Ajabbi.

Ajabbi is a bootstrapped, mission-led organisation with no investors, with all future profits going to a yet-to-be-established non-profit foundation to fund open research, science communication and support open source and the Pipi user community.

Pipi is designed to help make massive enterprise systems for socially useful critical infrastructure self-managing. Over time, it will extend to many platforms, in many human languages and writing systems.

There will be many ways to participate in Ajabbi, including but not limited to;

  • Volunteers
  • Visitors
  • Contractors
  • Interns
  • Staff
  • Research grantees

Ajabbi will need all kinds of people, with many different skills, from many countries. Pipi systems are novel, non-LLM, and we expect to solve unexpected hard problems by working as a team and just figuring it out. The work will be important, very difficult and interesting.

To register your interest, use the Google Form below. You can share this link with others. Everyone is welcome.

By , No comments:
Tags , ,

The Roadmap of Mathematics for Machine Learning

Mike's Notes

A wonderful article by Tivadar Danka. The Palindrome is a fabulous resource for learning mathematics and machine learning.

Resources

References

  • The Mathematics of Machine Learning by Tivadar Danka.
  • Linear Algebra Done Right by Sheldon Axler.
  • The Calculus book by Gilbert Strang.
  • Introduction to Probability by John Tsitsiklis.

Repository

  • Home > Ajabbi Research > Library > Subjects > Mathematics
  • Home > Ajabbi Research > Library > Subscriptions > The Palindrome
  • Home > Handbook > 

Last Updated

22/09/2026

The Roadmap of Mathematics for Machine Learning

By: Tivadar Danka
The Palindrome: 06/08/2026

Tivadar Danka: Hungarian mathematician and machine learning enthusiast.

...

A complete guide to linear algebra, calculus, and probability theory

Understanding the mathematics behind machine learning algorithms is a superpower.

If you have ever worked on a real-life problem, you probably experienced that being familiar with the details can go a long way if you want to move beyond baseline performance. This is especially true when you want to push the boundaries of state-of-the-art.

However, most of this knowledge is hidden behind layers of advanced mathematics. Understanding methods like stochastic gradient descent may seem difficult, as they are built on top of multivariable calculus and probability theory.

With proper foundations, though, most ideas can be seen as quite natural. If you are a beginner and don’t necessarily have formal education in higher mathematics, creating a curriculum for yourself is hard. In this post, my goal is to present a roadmap that takes you from absolute zero to a deep understanding of how neural networks work.

To keep things simple, the aim is not to cover everything. Instead, we will focus on getting our directions. This way, you will be able to study other topics without difficulties, if need be.

Instead of reading through in one sitting, I recommend using this article as a reference point through your studies. Go deep into a concept that is introduced, then check the roadmap and move on. I firmly believe that this is the best way to study: I will show you the road, but you must walk it.

Machine learning is built upon three pillars: linear algebra, calculus, and probability theory.

Here’s the full roadmap for you.

Linear algebra is used to describe models, calculus is to fit the model to the data, and probability theory is to tie it all together by providing a theoretical framework for making predictions under uncertainty.

What follows is a ~4000 words-long deep dive into all of these, where we’ll walk through the main topics step by step.

If you are interested in the mathematics of machine learning, this is the article you want to read.

A quick note. This post is way too long for email. For the best experience, read the post in your browser or in the Substack app.

Linear algebra

Predictive models such as neural networks are essentially functions that are trained using the tools of calculus. However, they are described using linear algebraic concepts, such as matrix multiplication.

If you are a machine learning engineer working on real-life problems, linear algebra is the most important topic for you, and mastering it will put you above the playing field.

Let’s dive deep!

Vectors and vector spaces

To have a good understanding of linear algebra, I suggest starting with vector spaces.

The textbook definition is intimidating and abstract, so let’s talk about a special case first. You can think of each point in the plane as a tuple x = (x₁, x₂), represented by an arrow pointing from the origin to (x₁, x₂).

You can add these vectors together and multiply them with scalars. Algebraically, it simply goes as

but it’s easier to visualize:

The Euclidean plane is the prototypical model of a vector space. Tuples of n elements form n-dimensional vectors, making up the Euclidean space.

In general, a set of vectors V is a vector space over the real numbers if you can add and scale vectors in a straightforward way.

When thinking about vector spaces, it helps to model them as tuples in Euclidean space mentally.

Normed spaces

When you have a good understanding of vector spaces, the next step is to understand how to measure distance in vector spaces.

By default, a vector space in itself gives no tools for this. How would you do this on the plane? You have probably learned that there, we have the famous Euclidean norm defined by.

Although the vector notation and the square root symbol make this feel intimidating, the magnitude is just the Pythagorean theorem in disguise.

This can be generalized further: in three dimensions, the Euclidean norm is the repeated application of the Pythagorean theorem.

This is a special case of a norm. In general, a vector space V is normed if there is a function ‖ ⋅ ‖: V → [0, ∞) such that

where x and y are any two vectors.

Again, this might be scary, but this is a simple and essential concept. There are a bunch of norms out there, and the most important is the p-norm family, defined for any p ∈ [0, ∞) by

(with p = 2 giving the Euclidean norm) and the supremum norm

Norms can be used to define a distance by taking the norm of the difference:

The 1-norm is called the Manhattan norm (or taxicab norm), because the distance between two points depends on how many “grid jumps” you have to perform to get from x to y.

Sometimes, like for p = 2, the norm comes from a so-called inner product, which is a bilinear function 〈 ⋅, ⋅ 〉: V × V → [0, ∞) such that

A vector space with an inner product is called an inner product space. An example is the classical Euclidean product

On the other hand, every inner product can be turned into a norm by

When the inner product for two vectors is zero, we say that the vectors are orthogonal to each other. (Try to come up with some concrete examples on the plane to understand the concept more deeply.)

Basis and orthogonal/orthonormal basis

Although vector spaces are infinite (in our case), you can find a finite set of vectors that can be used to express all vectors in the space. For example, on the plane, we have

where

This is a special case of a basis and an orthonormal basis.

In general, a basis is a minimal set of vectors v₁, v₂, ..., vₙ ∈ V such that their linear combinations span the vector space:

A basis always exists for any vector space. (It may not be a finite set, but that shouldn’t concern us now.) Without a doubt, a basis simplifies things greatly when talking about linear spaces.

When the vectors in a basis are orthogonal to each other, we call it an orthogonal basis. If each basis vector’s norm is 1 for an orthogonal basis, we say it is orthonormal.

Linear transformations

The key objects related to vector spaces are linear transformations. If you have seen a neural network before, you know that the fundamental building blocks are layers of the form f(x) = σ(Ax + b), where A is a matrix, b and x are vectors, and σ is the sigmoid function. (Or any activation function, really.) Well, the part Ax is a linear transformation.

In general, the function L: V → W is a linear transformation between vector spaces V and W if

holds for all x, y in V, and all real number a.

To give a concrete example, rotations around the origin in the plane are linear transformations.

Undoubtedly, the most crucial fact about linear transformations is that they can be represented with matrices, as you’ll see next in your studies.

Matrices and their operations

If linear transformations are clear, you can turn to the study of matrices. (Linear algebra courses often start with matrices, but I would recommend it this way for reasons to be explained later.)

The most important operation for matrices is matrix multiplication, also known as the matrix product. In general, if A and B are matrices defined by

then their product can be obtained by

This might seem difficult to comprehend, but it is pretty straightforward. Take a look at the figure below, demonstrating how to calculate the element in the 2nd row, 1st column of the product.

The reason why matrix multiplication is defined the way it is because matrices represent linear transformations between vector spaces. Matrix multiplication is the composition of linear transformations.

Determinants

In my opinion, determinants are hands down one of the most challenging concepts to grasp in linear algebra. Depending on your learning resource, it is usually defined by either a recursive definition or a sum that iterates through all permutations. None of them is tractable without significant experience in mathematics.

To understand this concept, watch this video below. Trust me, it is magic.

To summarize, the determinant of a matrix describes how the volume of an object scales under the corresponding linear transformation. If the transformation changes orientations, the sign of the determinant is negative.

You will eventually need to understand how to calculate the determinant, but I wouldn’t worry about it now.

Eigenvalues, eigenvectors, and matrix decompositions

A standard first linear algebra course usually ends with eigenvalues/eigenvectors and some special matrix decompositions like the Singular Value Decomposition.

Let’s suppose that we have a matrix A. The number λ is an eigenvalue of A if there is a vector x (called an eigenvector) such that Ax = λx holds. In other words, the linear transformation represented by A is a scaling by λ for the vector x. This concept plays an essential role in linear algebra. (And practically in every field that uses linear algebra extensively.)

At this point, you are ready to familiarize yourself with a few matrix decompositions. If you think about it for a second, what type of matrices are the best from a computational perspective? Diagonal matrices! If a linear transformation has a diagonal matrix, it is trivial to compute its value on an arbitrary vector:

Most special forms aim to decompose a matrix A into a product of matrices, where hopefully at least one of the matrices is diagonal. Singular Value Decomposition, or SVD in short, the most famous one, states that there are special matrices U, V, and a diagonal matrix Σ such that A = U Σ V holds. (U and V are so-called unitary matrices, which I don’t define here; suffice to know that it is a special family of matrices.)

SVD is also used to perform Principal Component Analysis, one of the simplest and most well-known methods for dimensionality reduction.

Further study

Linear algebra can be taught in many ways. The path I outlined here was inspired by the textbook Linear Algebra Done Right by Sheldon Axler. For an online lecture, I would recommend the Linear Algebra course from MIT OpenCourseWare, an excellent resource.

Here are all of my articles on the topic:

  • Matrices and graphs
  • How to measure the angle between two functions
  • The unreasonable effectiveness of orthogonal systems

Calculus

Calculus is the study of differentiation and integration of functions. Essentially, a neural network is a differentiable function, so calculus will be a fundamental tool to train neural networks, as we will see.

To familiarize yourself with the concepts, you should make things simple and study functions of a single variable for the first time. By definition, the derivative of a function is defined by the limit

where the ratio for a given h is the slope of the line between the points (x, f(x)) and (x+h, f(x+h)).

In the limit, this is essentially the slope of the tangent line at the point x. The figure below illustrates the concept.

Differentiation can be used to optimize functions: the derivative is zero at local maxima or minima. (However, this is not true in the other direction; see f(x) = x³ at 0.)

Points where the derivative is zero are called critical points. Whether a critical point is a minimum or a maximum can be decided by looking at the second derivative:

There are several essential rules regarding differentiation, but probably the most important is the so-called chain rule:

which tells us how to calculate the derivative of composed functions.

Integration is often called the inverse of differentiation. This is true because if f is the derivative of F, that is, F′(x) = f(x), then

holds. (If f is an integrable function.)

The integral of a function can also be thought of as the signed area under the curve:

Integration itself plays a role in understanding the concept of expected value. For instance, quantities like entropy and Kullback-Leibler divergence are defined in terms of integrals.

Further study

I would recommend the Single Variable Calculus course from MIT. (In general, online courses from MIT are always excellent learning resources.) If you are more of a book person, there are many textbooks available. The Calculus book by Gilbert Strang, which accompanies the previously mentioned course, is again a great resource, available completely free of charge.

Here are some of my articles on the topic:

  • Why does gradient descent work?
  • The history of trigonometric functions
  • The fascinating story of the exponential function

Multivariable calculus

This is the part where linear algebra and calculus come together, laying the foundations for the primary tool to train neural networks: gradient descent. Mathematically speaking, a neural network is simply a function of multiple variables. (Although the number of variables can be in the millions.)

Similar to univariate calculus, the two main topics here are differentiation and integration. Suppose we have a function f: ℝⁿ → ℝ, mapping vectors to real numbers.

In two dimensions (that is, for n = 2), you can imagine its plot as a surface. (Since humans don’t see higher than three dimensions, it is hard to visualize functions with more than two real variables.)

Differentiation in multiple variables

In a single variable, the derivative was the slope of the tangent line. How would you define tangents here? A point on the surface has several tangents, not just one. However, there are two special tangents: one is parallel to the x-z plane, while the other is parallel to the y-z plane. Their slope is determined by the partial derivatives, defined by

That is, you take the derivative of the functions obtained by fixing all but one variable. (The formal definition is identical for ≥ 3 variables, just more complicated notation.)

Tangents in these special directions span the tangent plane.

The tangent plane

The gradient

There is another special direction: the gradient, which is the vector defined by

The gradient always points to the direction of the largest increase! So, if you would take a tiny step in this direction, your elevation would be the maximal among all the other directions you could have chosen. This is the basic idea of gradient descent, an algorithm used to maximize functions. Its steps are the following.

  1. Calculate the gradient at the point x₀, where you currently are.
  2. Take a small step in the direction opposite to the gradient to arrive at the point x₁. (The step size is called the learning rate.)
  3. Go back to Step 1 and repeat the process until convergence.

Of course, there are several flaws in this basic algorithm, which has been improved several times over the years. Modern gradient descent-based optimizers employ various techniques, such as adaptive step size, momentum, and other methods, which we will not detail here.

Calculating the gradient in practice is difficult. Functions are often described by the composition of other functions, for instance, the familiar linear layer

where A is a matrix, b and x are vectors, and σ is the sigmoid function. (Of course, there can be other activations, but we shall stick to this for simplicity.) How would you calculate this gradient? At this point, it is not even clear how to define the gradient for vector-vector functions such as this, so let’s discuss!

The function g(x): ℝⁿ → ℝᵐ can always be written in terms of vector-scalar functions like

The gradient of g is defined by the matrix whose k-th row is the k-th component’s gradient. That is,

This matrix is called the total derivative of g.

In our example f(x) = σ(Ax + b), things become a bit more complicated because it is the composition of two functions:

  1. l(x) = Ax + b,
  2. and σ(x),

defined by applying the univariate sigmoid componentwise. The function l can be decomposed further to m functions mapping from the n-dimensional vector space to the space of real numbers:

where

If you calculate the total derivative, you’ll see that

This is the chain rule for multivariate functions in its full generality. Without it, there would be no easy way to calculate the gradient of a neural network, which is ultimately a composition of many functions.

Higher-order derivatives

Similarly to the univariate case, the gradient and derivatives play a role in determining whether a given point in space is a local minimum or maximum. (Or neither.) To provide a concrete example, training a neural network is equivalent to minimizing the loss function on the parameters’ training data. It is all about finding the optimal parameter configuration w for which the minimum is attained:

where N: ℝⁿ → ℝᵐ and l: ℝᵐ → ℝ are the neural network and the loss function, respectively.

For a general differentiable vector-scalar function of n variables, there are n² second derivatives, forming the Hessian matrix

In multiple variables, the determinant of the Hessian takes the role of the second derivative. Similarly, it can be used to determine whether a critical point (i.e., where all the derivatives are zero) is a minimum, maximum, or just a saddle point.

Further study

There are lots of fantastic online courses on multivariable calculus. I have two specific recommendations:

  • MIT multivariable calculus
  • Khan Academy on multivariable calculus

Now we are ready to take on the final subject: probability theory!

Probability theory

Probability theory is the mathematically rigorous study of chance, fundamental to all fields of science.

Setting exact definitions aside for now, let’s ponder a bit about what probability represents. Let’s say I toss a coin, with a 50% chance (or 0.5 probability) of it being heads. After repeating the experiment 10 times, how many heads did I get?

If you have answered 5, you are wrong. Heads being 0.5 probability doesn’t guarantee that every second throw is heads. Instead, what it means that if you repeat the experiment n times where n is a really large number, the number of heads will be very close to n/2.

Besides the basics, there are some advanced things you need to understand, first and foremost, expected value and entropy. But let’s not get ahead of ourselves! First, we have to understand what probability is!

The concept of probability

First of all, probability is a function that renders a numeric value between 0 and 1 to events. Events are represented by sets, which are subsets in the event space, denoted by Ω.

There are two defining properties of probability:

  1. the probability of the entire event space is 1,
  2. and the probabilities of disjoint events can be summed up.

In terms of mathematical formulas:

Set operations can also be translated to the language of events: A ∪ B means that either A or B occurs, while A ∩ B means that both occur.

One of the most important concepts in probability theory is conditional probability, which studies probabilities in the context of observations. By definition, it is given by the ratio of the probability of both events occurring and the probability of the observed event.

One simple example: what is the probability of a given email being spam, if it contains the word “deal”? Observing keywords like the mentioned “deal” increases the probability of the email being spam.

In certain practical scenarios, we only know P(A | B), but we want to estimate P(B | A). This is what Bayes’ theorem is for, expressing one in terms of the other.

In English, Bayes’ theorem shows us how to update our priors in terms of the likelihood.

Expected value

Suppose that you play a game with your friend. You toss a classical six-sided dice, and if the outcome is 1 or 2, you win 300 bucks. Otherwise, you lose 200. What are your average earnings per round if you play this game long enough? Should you even be playing this game?

Well, you win 100 bucks with probability 1/3, and you lose 200 with probability 2/3. That is, if X is the random variable encoding the result of the dice throw, then

This is the expected value, that is, the average amount of money you will receive per round in the long run. Since this is negative, you will lose money, so you should never play this game.

Generally speaking, the expected value is defined by

for discrete random variables and

for real-valued continuous random variables.

In machine learning, loss functions for training neural networks are expected values in one way or another.

Law of large numbers

People often falsely attribute certain phenomena to the law of large numbers. For instance, gamblers who are on a losing streak believe that they should soon win because of the law of large numbers. This is totally wrong. Let’s see what this is really!

Suppose that X, X₁, X₂, ... are random variables representing the independent repetitions of the same experiment. (Say, rolling a dice or tossing a coin.)

The (strong) law of large numbers states that

holds with probability one; that is, the average of the outcomes in the long run equals the expected value.

An interpretation is that if a random event is repeated enough times, individual results might not matter. So, if you are playing in a casino with a game that has negative expected value (as they all do), it doesn’t matter that you win occasionally. The law of large numbers implies that you will lose money.

To get a little bit ahead, LLN is going to be essential for stochastic gradient descent.

Information theory

Let’s play a game. I have thought of a number between 1 and 1024, and you have to guess it. You can ask questions, but your goal is to use as few questions as possible. How much do you need?

If you play it smart, you will perform a binary search with your questions. First, you may ask: is the number between 1 and 512? With this, you have cut the search space in half. Using this strategy, you can figure out the answer in log₂(1024) = 10 questions.

But what if I didn’t use the uniform distribution when picking the number? For instance, I could have used a Poisson distribution.

Here, you would probably need fewer questions because you know that the distribution tends to concentrate around specific points. (Which depends on the parameter.)

In the extreme case, when the distribution is concentrated on a single number, you need zero questions to guess it correctly. Generally, the number of questions depends on the information carried by the distribution. The uniform distribution contains the least amount of information, while singular ones are pure information.

Entropy is a measure that quantifies this. It is defined by

for discrete random variables and

for continuous, real-valued ones. (The base of the logarithm is usually 2, e, or 10, but it doesn’t really matter.)

If you have ever worked with classification models before, you probably encountered the cross-entropy loss, defined by

where P is the ground truth (a distribution concentrated to a single class), while the hatted version represents the class predictions. The cross-entropy loss measures how much “information” the predictions have compared to the ground truth. When the predictions match, the cross-entropy loss is zero.

Another frequently used quantity is the Kullback-Leibler divergence, defined by

where P and Q are two probability distributions. This is essentially cross-entropy minus the entropy, which can be thought of as quantifying the difference between the two distributions. This is useful, for instance, when training generative adversarial networks. Minimizing the Kullback-Leibler divergence guarantees that the two distributions are similar.

Further study

Here, I would again recommend an online course from MIT, which covers all the fundamentals and some advanced concepts. Check out Introduction to Probability by John Tsitsiklis!

Some of my articles about probability:

  • The Law of Large Numbers
  • Is probability frequentist or Bayesian?
  • Probabilities, densities, and distributions
  • What's the meaning of the expected value?

P.S. If you want a single resource with a full breakdown of this entire roadmap, consider buying my book, The Mathematics of Machine Learning.

By , No comments:
Tags , , , ,
Subscribe to: Posts (Atom)