When life gives you lemons, write better error messages

Mike's Notes

I found this article on Medium, which gives me somewhere to start on getting the Content Management System to store and generate Error Messages.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

18/05/2025

When life gives you lemons, write better error messages

By: Jenni Nadler
Wix UX on Medium: 12/09/2022

When it comes to error handling, it truly is a team sport

Error messages are part of our daily lives online. Every time a server is down or we don’t have internet, or we forget to add some info in a form, we get an error message. “Something went wrong” is the classic. But what went wrong? What happened? And, most importantly, how can I fix it?

We encounter error messages all the time, but how often do they actually help us understand what went wrong and how to fix it?

About a year ago at Wix, we abruptly realized that, too often, we were not giving users the answers to these questions. When we got this wake-up call, we felt compelled to act swiftly, and not just to address the one error message that woke us up.

Welcome, folks, to Errorgate 2021.

Or, that time we changed thousands of error messages across Wix in just a month.

To complete this effort, we first had to define for ourselves what counted as a bad error message and what counted as a good error message.

What makes a bad error message

This is an example of a bad error message. It uses an inappropriate tone, passes the blame, speaks in technical jargon and is too generic.

Inappropriate tone: Imagine a doctor performing a procedure and then suddenly saying “Oops! Something went wrong…” That is the last thing anyone wants to hear when the stakes are high, whether it’s surgery or someone’s source of income. That is not the time to be cutesy or fluffy. We want to show the users that we know it’s serious and we understand it’s important to them.

Technical jargon: Even in today’s world of user-centered design, technical jargon still sneaks its way into error messages. You couldn’t fetch my data? My credentials were denied? What? The technical stuff is not important to the user, they just want to know what went wrong and how to fix it.

Passing the blame: Try to focus on the problem, rather than the action that led to the problem. We don’t want to shame users, even if something they did is why they’re seeing a certain error message.

We also made the decision not to pass blame on to third parties because it makes us look unprofessional, even if it would have taken some of the burden off of Wix. The user came to Wix as a trusted platform; they don’t want to think about other platforms. While we can say something like, “We’re having trouble connecting to ___”, we wouldn’t say something like, “___ isn’t responding right now.”

Generic for no reason: Sometimes we don’t know what caused the error… but sometimes we do. If we know what caused it and we’re not telling them, we’re doing our users the ultimate disservice.

What makes a good error message

An error pop-up demonstrating a good error message with each section highlighted to show why it’s better than the bad example. “Unable to connect your account (explains what happened). Your changes were saved (provides reassurance), but we could not connect account due to a technical issue on our end (explains also why the error happened). Please try connecting again (displays empathy and helps the user fix the issue). If the issue keeps happening, contact Customer Care (gives the user a way out


This is an example of a good error message. It explains what happened and why, provides reassurance, is empathetic, helps the user fix the issue and gives the user a way out.

Say what happened and why: Make it super clear what did or didn’t happen. This can be done with a combination of visuals and text. Explain why the user got this error, even if the only explanation is that there was a technical issue. At Wix, we made the decision to say “an issue on our end” if we have the space, to really reiterate that it’s not the user’s fault.

Provide reassurance: Where possible, let them know what was not affected by the error. For example, were their changes still saved as a draft, even though their email wasn’t sent?

Be empathetic: While we don’t want to be overly apologetic, we decided that we did still want to use “please” if the situation warrants it. Maybe it’s a really dire situation, or it’s something that we absolutely can’t help the user solve. In that case, we might use “please” to empathize even more.

Help them fix it: Tell them exactly what to do if there’s a way to possibly fix it. Short on space? Send them to a knowledge base article with a descriptive link like, “Learn how to resolve this” or “How do I fix this?”

Always give a way out: If they can’t fix the problem, or if it’s possible the issue could keep happening, provide them with a way to contact Customer Care.

Now that we had defined what makes a good or a bad error message, we had to start getting rid of the bad ones.

How we tackled removing bad error messages

We did a search in our content management system and found that there were 7,643 keys with the word “error” in the key or value. That’s 7,643 pieces of content that–at the very least–needed to be reviewed.

The task seemed monumental.

But we did it. We reviewed every single piece of content related to errors and decided if it was relevant for this effort. Once we had a list of all the errors we considered “generic” or “not helpful”, we sent everything to developers.

This was just one of the Monday.com boards that we used to categorize every single piece of content related to errors. Boards like these helped us set priorities, due dates and keep all disciplines in the loop.

Developers went message by message and mapped where each was being triggered in the code. They looked at what was causing the message to show, how frequently it was occurring, and what could be done to resolve the issue.

Based on that error mapping, the product managers, UX designers, and writers sat down and came up with solutions. We started by transferring everything from a spreadsheet to a Monday board, where we could easily track the status of things and what needed to be done. Sometimes, it was just a simple content change. In other cases, it required brand new error messages. And in lots of other instances, there was additional development work that needed to be done to fix things behind the scenes.

Then, we prioritized which errors to work on first. To set priorities, we focused on how often the error was happening and if it blocked the user from completing the flow. After that, we set milestones of one to four weeks, so that things didn’t fall by the wayside.

What we learned

There’s a difference between generic and unclear messages. While there were certainly a lot of generic “Something went wrong” messages, there were also a lot of unclear messages. These are just as bad as generic messages, and deserve the same amount of attention.

A generic message next to an unclear message. Generic message: “Something went wrong and this action could not be completed.” Unclear message: “Make sure you allow the requested permissions and try again.”

An example of a generic message compared to a message that is unclear. In the generic message, we’re simply not telling the user anything other than something went wrong. In the unclear message, we tried to explain what went wrong, but it used confusing language.

It’s not a content issue most of the time. Avishai Abrahami, our CEO and the reason this project got started, put it best in his email to all employees. “Generic errors are the result of bad development and product. … We must all care about it together.”

Truly everyone in Wix had to come together across all disciplines to fix these messages. Developers had to investigate and map. Product managers had to prioritize and create tasks. Designers had to provide new designs for new flows. And we, the UX writers, had to write and rewrite thousands of error messages.

We should be asking more questions. It used to be really common for a developer to say to us, “Hey, we need a generic error message here. Can you add one?” And we would say yes, thinking it would be a fallback or rare message. We didn’t often stop to ask questions like, “Why are users seeing this?” and “What is happening in the background?”

We missed a learning opportunity. Unfortunately, we were reactive instead of proactive here. If this effort had been strategically planned, it could have been an amazing learning opportunity for junior writers in particular. Instead, we were scrambling to write and rewrite messages without much strategic thought.

We were being a bad friend. At Wix, we have the mantra, “Write it like you’re talking to a friend.” We really believe in empathizing with the user, and being a friend with them throughout their process. But it turns out that we were more like that friend who loves to gossip, but doesn’t pick up the phone when life gets hard. That is not the friend we want to be, so we had to really dig deep and admit that we weren’t doing the best we could.

When we work together, we build better products. It’s cheesy, but it’s true.

What we’ve changed in our process

Established a cross-functional team to focus on error handling. This team is made up of senior product managers, frontend and backend developers, UX designers and UX writers. Their goal is to make sure proper error handling is part of the product life cycle, not an afterthought.

View it as a shared responsibility. Everyone is responsible for making sure we’re handling errors properly. Product managers are expected to place more emphasis on errors and edge cases, not just happy flows. Developers are expected to investigate and document errors according to platformized guidelines. Data scientists are expected to do better analysis on errors so we can track the events properly.

Review errors one month after launch. Sometimes, especially if it’s a brand new product, we don’t even know what errors to expect. So we might have to launch with generic errors, but now we have a procedure where we review the errors occurring one month after launch. This allows us to see what really are the biggest errors and write content specifically for those.

Ongoing review process. As writers, we know everything can always be optimized. So we’re constantly reviewing our errors, even the ones we just updated recently.

UX writers are empowered to challenge generic errors. In case a product manager or developer ever says, “Let’s just use this generic error message in all cases”, we now have the power to say no. The CEO of the company has said generic errors are not acceptable, so we’re not going to write them without more investigation and understanding of the problem. The power lies with us!

All in all, we changed thousands of error messages by working together with our colleagues. It was hard work and we all had a drink or two at the end of it. But it was the right thing to do for our users, and the only way to truly live up to our value of putting the user first.

Edited by Dan Raz. Graphics by Yansou Girard.

SNOMED Release Notes

Mike's Notes

I'm rather impressed with the way SNOMED creates its release notes. The URL to the resource is obvious, and the TOC provides in-page navigation to the main content.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

18/05/2025

SNOMED Release Notes

By: Mike Peters
On a Sandy Beach: 19/08/2024

Mike is the inventor and architect of Pipi and the founder of Ajabbi.

Example

Page At A Glance

1. Introduction
1.1. Background
1.2. Purpose
1.3. Scope
1.4. Audience
1.5. Important Information
2. Content Development Activity
2.1. Summary
2.2. Quality Initiative
2.2.1. Update for Pathological Process and Infectious Process
2.2.2. Update Hierarchy 90128006|Photosensitivity (finding)|
2.2.3. Update 230131008|Graft complications (disorder)|
2.2.4. Update Hierarchy 278854009|Insertion of hormone implant (procedure)|
2.3.  Body Structure 
2.3.1. Update Lateralizable Body Structures
2.3.2. SEP and Laterality Anatomy Reference Sets
2.4. Clinical Finding
2.4.1. Inactivation of Mechanical Complication of <x> Disorder Concepts
2.5. Procedure
2.5.1. Radiographic Imaging
2.5.2. Remodel Hierarchy 51302004 |Turbinectomy (procedure)|
2.5.3. Remodel Hierarchy 359890009|Fundoplication (procedure)|
2.6. Pharmaceutical/Biological Product
2.6.1. Grouping Concepts for Therapeutic Roles in the Medicinal Product Hierarchy
2.6.2. Update Pharmaceutical Dose Forms with Two or More Intended Sites
2.7. Collaboration/Harmonization Agreements
2.7.1. Convergent Medical Terminology (CMT)
2.7.2. Orphanet
2.7.3. Cancer Synoptic Reporting
2.7.4. International League Against Epilepsy (ILAE)
2.7.5. Gravity Project
2.8. Internal Quality Improvement
2.8.1. Machine Readable Concept Model (MRCM) Changes
2.8.2. OWL Axiom Expressions for Annotation Properties
2.9. SNOMED CT derived products
2.9.1. ICD-10 map
2.9.2. Content Development Activity Summary
2.9.3. SNOMED CT to OWL conversion and classification
3. Technical notes
3.1. Known Issues
3.2. Resolved Issues
3.3. Technical updates
3.3.1. RF2 package format
3.3.2. Changes to the International Edition package format
3.3.3. Changes to the Annotations Refset format
3.3.4. Changes to the RefsetDescriptor records
3.3.5. Early visibility of impending changes in the upcoming 2024 Monthly International Edition releases
3.3.6. Document links

How Stripe Built a Writing Culture

Mike's Notes

Stripe’s Documentation Manager shares how the company built a culture where writing is second nature. The article is copied from SLAB.

The draft Ajabbi Design System Style Guide is based on the open-source MailChimp Style Guide published on Slab. Many sample or open-source complex documents are available on Slab.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

18/05/2025

How Stripe Built a Writing Culture

By: RC Victorino
SLAB: 02/09/2020

For a company whose product focuses on numbers, Stripe has built an enviable writing culture.

Their quarterly software engineering magazine (Increment) and publishing imprint (Stripe Press) are impressive on their own. But it’s Stripe’s internal writing culture that differentiates the company. Whether it’s an email to a colleague or a memo to an entire team, Stripe employees strive to write with excellence.

Their strong writing culture benefits the organization in several areas:

  1. Time efficiency. Sharing ideas through writing eliminates the need for repetitive verbal updates to disseminate ideas and information.
  2. Knowledge sharing. Documenting important ideas forces clarity of thought and makes information more accessible to everyone in the company, versus slide decks that are ephemeral and require less rigor of thought.
  3. Communication. Clear writing requires clear thinking, meaning employees invest more time shaping their ideas before sharing them.

“Writing forces you to structure your thoughts in a manner just not possible when you verbalize it. When I write, I have to offer structured, precise thoughts.”

Dave Nunez
Documentation Manager at Stripe

The result is that Stripe’s internal and external documentation is a central pillar of the company’s reputation and brand.

How did this happen? We sat down with Stripe’s Documentation Manager, Dave Nunez, to learn more about what it takes to build a strong writing culture.

Leaders must demonstrate quality writing

Getting employees to write more frequently shouldn’t be your ultimate goal. Getting them to write more proficiently should be.

One of the most effective ways to encourage proficient writing is to demonstrate it.

“The first emails I saw from our CEO [Patrick Collison] literally had footnotes,” Nunez recalls. “He structured his emails to be like research papers and put the peripheral information at the bottom so as not to detract from the core information.”

Today, footnotes are a common component of internal emails at Stripe. The CEO set the expectation; employees strive to uphold it. But Collison’s approach to writing did more than demonstrate the value of a footnote. It’s understood that at Stripe:

  1. Writing matters. Make it count.
  2. Everyone’s time is valuable. Make the effort to put together coherent ideas in writing.
  3. The clearer the writing, the clearer your message and intent will be.

Collison isn’t alone in his passion for quality writing. The modus operandi for leadership communications across Stripe is carefully structured narrative documents and emails. You’re far more likely to read a narrative memo during a Stripe project kickoff meeting than to sit through a PowerPoint presentation.

“From leadership on down, we default to writing,” Nunez said. “We don’t really have slide decks.”

How can you lead by example to foster a writing culture? Here are two ideas inspired by the Stripe playbook.

1. Exemplify quality writing in everything you and your leadership team share.

Footnotes might not be right for every company, but strategies anyone can adopt include citing reputable sources, being economical with your words, and ensuring your writing is free from grammar and spelling errors.

As a leader, consider having someone else review your writing for clarity and readability. Then, let your team know you had your writing reviewed. This transparency showcases just how committed you are to increasing the impact of your words.

“You wouldn’t ship code without having it reviewed; your words are just as important,” says Nunez.

2. Make writing the default method of sharing knowledge.

Eschew slide decks for narrative memos. Share new ideas through carefully crafted emails.

When an employee pitches an idea to you, ask them to expand on it in writing.

Show them you value the effort required to convey one’s thoughts into writing by explicitly granting them the time necessary to think through and process their thoughts into writing.

“I’m only a month into my time at Stripe, but I’ve never encountered a tech company of this size where writing is such a center of gravity.”

Shaun Young
Editorial at Stripe

Give teammates a starting point with sample docs

Nunez and his team create sample docs that other teams can use as inspiration for their own documents.

For example, his team published a detailed guide on the life of a Stripe charge. They walked through every step of the process, from the point of sale with the customer, to the back end, bank transactions, and more.

While the content itself was specific to a Stripe charge, the document serves as a valuable reference to other teams on how to write a guide. Teams use this document to understand what type of language to use, to see what types of visuals are most effective, and to learn how to structure their writing for better comprehension (such as when and where to use subheads and lists).

Stripe has similar sample docs for READMEs, runbooks, and FAQs. Nunez believes these sample documents are more useful than fill-in-the-blank templates, because they provide readers more context and content to work with.

“We create docs that offer some of the basics, so engineers aren’t forced to stare at a blank page — which can be terrifying,” Nunez says.

Not every company has a dedicated documentation department. But creating sample documents doesn’t have to be an arduous task. Identify the types of documents most teams would want to produce — then have your most proficient writers create ambitious documents that can be later used as reference documents.

Know when to standardize and when to give autonomy

Standardizing how your team documents shared knowledge ensures content is easy to understand and streamlines the writing process.

But it’s neither scalable nor empowering to standardize everything.

It’s not scalable because it would take constant oversight to ensure every document met a specific format. Few companies have (or are willing to invest in) the resources for this kind of oversight.

It’s not empowering because what works for one team won’t necessarily work for another. Standardizing your entire documentation process robs teams of creating an experience that works best for them.

Stripe toes the line between control and autonomy by establishing a standardized approach for high-leverage documents only — those with a broader audience (like a document intended for multiple teams) or significant implications (like if it impacts business operations).

This approach ensures that the most widely read information receives the oversight it deserves, while teams still maintain autonomy over how they document knowledge most pertinent to them.

In contrast, each team at Stripe has far greater control over the look and feel of documents with a smaller audience and impact.

Standardization at Stripe isn’t represented by a series of templates employees plop their knowledge into like Mad Libs. Rather, standardization at Stripe is more about ensuring the clarity of the content and the reading experience live up to the company standard of quality writing. To ensure this, Nunez typically gets his team involved in reviewing these high-leverage documents before they’re shared.

Replicating this process is straightforward — establish criteria that differentiate high-leverage documents from low-leverage ones. For example, you could define high-leverage documents as:

  • Intended to be read by three or more teams
  • Contains information that will go unchanged for at least one year
  • Will impact general business operations

Documents designated as high-leverage should either follow a specific format, be reviewed by a dedicated team before publication, or both.

Make your documentation easy to read

The point of documenting something is to get others to read it. Without active readers inside your company, your writing culture will never flourish.

So, how do you get people to read what you wrote?

“I think the visual aspect is super important,” Nunez says, “because the first impression tells someone whether the content is approachable or not.”

A visually appealing document doesn’t always contain images and graphics. Nunez has seen beautiful documents that contain no visuals at all.

“You can look at the document and see, ok, this is super simple,” he says. “The intro is very short, there’s bulleted lists down here, and it just gets to the point.”

But, he admits, that’s rare. Visuals and diagrams simplify documents. They make them more approachable, which is why he suggests whenever you can replace text with diagrams, do it.

Diagrams or not, everyone on your team should consider the visual aspect of their writing. Here are a few items to consider:

  • Keep paragraphs short (3–4 sentences). If possible, make the first paragraph of a document 2–3 sentences.
  • Use subheads and bulleted lists to break up walls of text.
  • Consider your audience when writing. Some audiences value complex words — some don’t. The goal is to find the most compelling language for the audience you need to reach and act on your document.
  • Edit frequently. Edit your own work, and ask peers to edit it as well. Editing is the key to getting the best clarification of your idea.

Create a support system

Writing well is not supposed to be easy. Writing well requires more critical thinking (than, say, speaking off the cuff), which produces better results.

The payoff is worth the effort, which is why everyone on your team should strive to be strong writers.

However, sometimes the writing process can create significant barriers that prevent team members from ever sharing their ideas.

Employees who struggle with writing, or whose native language isn’t English, may feel less confident contributing their knowledge.

Nunez has seen this throughout his career — he worries that a company does itself a disservice when some employees don’t feel empowered to share their ideas in a writing-heavy environment.

To address this, he emphasizes the importance of onboarding classes for new hires focused on writing and documentation, as well as office hours, and self-service resources. This demystifies the documentation process — but it also emphasizes just how important documentation is to business operations from the outset.

One thing Nunez is starting to experiment with is pairing ESL employees with writing mentors. He’s done this informally over his career, but would love to see companies create more formal programs for writing mentorships.

“The idea here is that you come with your writing, and a judgment-free expert writer will help you as if they were your college English professor,’” Nunez says.

But even your team’s strongest writers need support. Nunez, for example, is the first to admit that his writing can become long-winded and confusing. So, he regularly shares his work with a handful of colleagues he trusts to offer kind but honest feedback.

Many others across Stripe have colleagues review their work, as well.

“Engineers do this with their code,” he says, “and we do it with our writing.”

Building your culture of writing and documentation

When your CEO uses footnotes in his email and your company publishes full-length books, it’s clear that writing matters.

But you don’t have to be Stripe to develop a culture that embraces writing and documentation. Lead by example; know when to standardize internal writing and when not to; make your documents easy to read; develop a support system that encourages and empowers everyone to write. These are the building blocks from which any company can build a culture where writing and documentation become second nature.

Health systems in New Zealand

Mike's Notes

One of the reasons for creating Pipi is to provide support for health systems.

I have followed SNOMED for several years and participated in an OMG health workflow effort during the COVID lockdown. I did not contribute much, but I learned by watching how a standard's body functions with people working remotely. Ken Rubin skillfully led the effort.

Locally, there is Health Information NZ (HINZ). This is the primary organisation involved in standards and interoperability. The NZ public health system has 3,000 applications that don't integrate and must be fixed. I recently signed up to use SNOMED.

My account at SNOMED CT has been approved.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

18/5/2025

Health systems in New Zealand

By: Mike Peters
On a Sandy Beach: 17/08/2024

Mike is the inventor and architect of Pipi and the founder of Ajabbi.

SNOMED

"SNOMED CT or SNOMED Clinical Terms is a systematically organized computer-processable collection of medical terms providing codes, terms, synonyms and definitions used in clinical documentation and reporting. SNOMED CT is considered to be the most comprehensive, multilingual clinical healthcare terminology in the world.[1][2] The primary purpose of SNOMED CT is to encode the meanings that are used in health information and to support the effective clinical recording of data with the aim of improving patient care. SNOMED CT provides the core general terminology for electronic health records. SNOMED CT comprehensive coverage includes: clinical findings, symptoms, diagnoses, procedures, body structures, organisms and other etiologies, substances, pharmaceuticals, devices and specimens.

SNOMED CT is maintained and distributed by SNOMED International, an international non-profit standards development organization, located in London, UK. SNOMED International is the trading name of the International Health Terminology Standards Development Organisation (IHTSDO), established in 2007" - Wikipedia

Webinar

Topic: Transforming healthcare interoperability with FHIR

12:30pm to 1:30pm, Wednesday 28 August 2024

Watch live or on demand

"FHIR has become a household name as the standard that has moved health data exchange into the modern era.

Experts from both sides of the Tasman will discuss the latest developments using FHIR for joined-up care and better user experience.

This will include an update on the HISO interoperability standards and supporting tools that are Health NZ's delivery priorities in 2024, including the NZ Health Terminology Service (NZHTS), SNOMED CT NZ Edition, and NZ Core Data for Interoperability (NZCDI).

Also hear about developments from the first FHIR Accelerator in Australia, as well as implementation of the New Zealand Patient Summary." - HiNZ

Flat vs. Deep Website Hierarchies

Mike's Notes

Navigation hierarchies are one of the challenges of creating the large documentation sets of Pipi. How deep should they go?

This is relevant to navigation in these pipiCMS publication classes.

  • pipiWiki (similar to MediaWiki)
  • pipiDocs (similar to javaDocs)
  • pipiLearn (similar to Coursera)

I found this excellent article from NN Group in 2013, partially reproduced below.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

18/05/2025

Flat vs. Deep Website Hierarchies

By: Kathryn Whitenton
NNGroup: 10/11/2013

"Summary:  Information can be organized in either flat or deep hierarchies; both have their advantages and pitfalls.

Virtually every website that has more than a few pages uses some structure for organizing the content. The most common (and most easily understood) structure is to categorize pages into groups, often with distinct subgroups. The end result is a hierarchy of content, a structure familiar to most of us from our interactions with organizations, families, and the natural world.

Decisions about exactly how content should be grouped can have dramatic consequences for how your site's structure works (or doesn't work) for users, but these nuances are difficult to understand at first glance. To analyze how a structure will work, we often need to create a visualization that shows a high-level view of how the different pages of a site relate to each other.

Consider these 2 structures: each represents the same amount of information, and shows a perfectly logical way of organizing the content for a website. Yet the end-user's experience of browsing these 2 hierarchies — even if they contain exactly the same information — will be very different. 

...

Representing Hierarchies in the Interface

With flat hierarchies, it's relatively easy for users to understand how any given page relates to other pages, as long as there are some visible navigation menus. But the deeper a hierarchy becomes, the more likely visitors are to become disoriented. For sites that are more than a few levels deep, breadcrumbs (which show a link for each level of the site from the homepage to the current page) can help users orient themselves and understand the site structure. Sitemaps are another useful way of helping users see the structure of a website.

Flat or Deep?

Should your website's hierarchy be flat or deep? Like most design questions, there's no single right answer, and going too far to either extreme will backfire. Flat hierarchies tend to work well if you have distinct, recognizable categories, because people don't have to click through as many levels. When users know what they want, simply get out of the way and let them find it. You can use card sorting to help decide how to structure the information in the way users think about it. Tree testing can help validate the structure by allowing users to attempt to find information in the proposed hierarchy.

But there are exceptions to every rule. In some situations, there are simply too many categories to show them all at one level. In other cases, showing specific topics too soon will just confuse your audience, and users will understand your offerings much better if you include some intermediate category pages to establish context.

Observing your users — via usability testing, analytics, and search logs — can help you understand what problems your audience needs to solve and how familiar users are with your content. This background knowledge is essential to achieving the right balance between a breadth and depth in your hierarchy."- NNGroup

Wolfram LLM Benchmarking Project

Mike's Notes

Gary Marcus mentioned this project in his newsletter today.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

18/05/2025

Wolfram LLM Benchmarking Project

By:
Wolfram: 15/08/2024

Using Wolfram Language to benchmark the performance of major LLMs.

As major users and analyzers of large language model (LLM) technology, we've been continually tracking the performance of LLMs. This project involves releasing our ongoing results, initially for a specific well-characterized code generation task.

The task consists of going from English-language specifications to Wolfram Language code. The test cases are exercises from Stephen Wolfram's An Elementary Introduction to the Wolfram Language. These exercises have been done online by millions of humans, and we've developed effective tools for determining functional correctness of code, which we're now applying to LLMs.

Raw Data (It is sortable on the original page)

Vendor Model Correct
Syntax
Correct
Functionality
OpenAI gpt-4OpenAI 99.8% 49.7%
OpenAI gpt-4 turboOpenAI 99.8% 46.2%
OpenAI gpt-4oOpenAI 100.0% 46.2%
Anthropic claude3-opusAnthropic 99.4% 44.4%
Anthropic claude3.5-sonnet Anthropic 99.7% 43.7%
Google gemini-1.5-pro-001Google 99.0% 40.8%
Meta Llama-3-70B-instructMeta 99.7% 39.6%
Google text-unicorn-001Google 99.6% 39.5%
OpenAI gpt-3.5 turboOpenAI 99.0% 38.5%
Mistral AI mistral-largeMistral AI 98.4% 38.2%
Mistral AI open-mixtral-8x22BMistral AI 98.2% 36.1%
Meta codellama-34bMeta 99.7% 36.1%
Mistral AI codestralMistral AI 97.5% 34.4%
Google gemini-1.5-flash-001Google 98.5% 33.8%
Meta codellama-13bMeta 98.7% 30.0%
Mistral AI mistral-smallMistral AI 97.5% 29.9%
Anthropic claude2.1Anthropic 96.6% 28.5%
Anthropic claude2Anthropic 87.3% 28.2%
Anthropic claude3-sonnetAnthropic 98.7% 27.8%
Mistral AI mistral-mediumMistral AI 88.4% 27.6%
DeepSeek deepseek-coder-7bDeepSeek 92.1% 27.3%
DeepSeek deepseek-coder-33bDeepSeek 92.8% 26.2%
Meta codellama-7bMeta 97.2% 26.0%
IBM granite-8BIBM 93.4% 25.4%
Google text-bison-002Google 98.1% 25.1%
Anthropic claude3-haikuAnthropic 98.4% 24.8%
Google code-bison-002Google 97.8% 24.6%
Google code-gecko-002Google 98.1% 24.5%
Google gemini-1.0-pro-002Google 94.5% 24.2%
DeepSeek deepseek-coder-6.7bDeepSeek 89.1% 22.8%
Meta Llama-3-8B-instructMeta 97.0% 22.1%
Google code-bison-001Google 95.8% 19.1%
Nous Research Nous-Hermes-2-Mixtral-8x7B-DPONous Research 79.9% 15.6%
OpenChat openchat3.5OpenChat 88.2% 15.2%
Microsoft Phi-3 miniMicrosoft 87.1% 14.1%
Aleph Alpha luminous-supremeAleph Alpha 81.8% 13.2%
Aleph Alpha luminous-supreme-control-20230501Aleph Alpha 86.8% 10.9%
Meta Llama2-13bMeta 91.4% 10.7%
DeepSeek deepseek-coder-1.3bDeepSeek 64.3% 10.7%
Aleph Alpha luminous-extendedAleph Alpha 77.7% 10.0%
Aleph Alpha luminous-supreme-control-20240215Aleph Alpha 56.3% 9.1%
Mistral AI mistral-tinyMistral AI 78.5% 8.2%
Aleph Alpha luminous-extended-control-20240215Aleph Alpha 69.2% 7.5%
Aleph Alpha luminous-baseAleph Alpha 62.4% 7.0%
Aleph Alpha luminous-base-control-20240215Aleph Alpha 76.3% 6.6%
Replit replit-code-v1_5-3bReplit 40.5% 4.2%
Meta llama2-7bMeta 26.4% 3.7%
Falcon LLM falcon-7bFalcon LLM 45.3% 3.3%

Exploring Autonomous Agents: A Semi-Technical Dive

Mike's Notes

An article by Dan Chen was published on Sequoia.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

18/05/2025

Exploring Autonomous Agents: A Semi-Technical Dive

By: Dan Chen
Sequoia: 11/04/2023

Agents are all the rage, but their planning capabilities currently outpace their ability to act reliably. How do we make them truly autonomous?

Over the past several weeks, autonomous AI agents have taken the world by storm. AutoGPT is one of the fastest growing Github repos in history, rocketing past PyTorch, every major Python web framework, and Python itself (sorry Guido) in number of stars. It’s led to some pretty sweet—albeit cherry-picked— demos and has captured the imagination of Twitter thought leaders and AI doomers alike. In a weird twist of fate, the popularity of agents has probably accelerated AI doomerism more than it has accelerated progress towards the superintelligence doomers fear.

An agent’s primary distinction from LLMs is that they run in a self-directed loop, largely augmented by a lightweight prompting layer and some kind of persistence or memory. The architecture varies from agent to agent, with some focused on task prioritization and others taking a more conversational roleplaying approach. The use cases are far reaching, from personal assistants to automated GTM teams. If you’re looking for an excellent primer and thesis on agents, look no further than my colleague Lauren Reeder’s post. And if you’re looking for the cutting edge for this family of agents, there’s LangChain’s recent post on Plan-and-Execute Agents.

Pretty exciting, but how far are we from this reality? In order to truly understand, I am a strong believer in getting to the nuts and bolts. This means getting your hands dirty with the code. So I sat down for an afternoon and took a quick look at the codebases for some popular agents. Here are some observations.

Implementation

The core autonomous agent loops are pretty straightforward. For instance, for AutoGPT, each agent has access to an initial prompt, a set of actions to execute, a history of messages, and a workspace where it can write executable code and files on the fly. An initial prompt might be something like “plan my daughter’s birthday—she really likes unicorns.” The agent then runs on a modified ReAct loop and is able to critique its own action through a second chat completion call with a special prompt that roughly boils down to: “review the proposed action and tell me whether it’s good or bad and why.” The agent calls the OpenAI chat completion endpoint with the following context: the original directive describing the goal of the agent (“plan a sweet unicorn-themed party”), some commands that represent procedural code the agent can run, as well as some short-term memory (the historical context of the agent’s prior reasoning and actions, up to the token limit). The text completion is formatted to a JSON dict of “thoughts” reasoning through the next action to take as well as the next selected action—potentially through another chat completion call. The agent pauses after each loop and waits for user input, which will be included in future context. The context and history can be persisted in storage: Pinecone, Redis, or more recently, Milvus or Weaviate—but honestly a JSON dump works just fine.

Some immediate observations: Although the demos can be astounding, agent implementations are pretty straightforward under the hood. AutoGPT is in essence a light prompting layer running on a recursive loop with persistent memory and which can write executable code on the fly. LangChain has a partial implementation of AutoGPT where they augmented their base agent with the optional human feedback component. It’s important to note: LangChain is a framework that allows for the implementation of various agents, including not just AutoGPT, but also BabyAGI and direct translations of existing research (e.g., ReAct, MRKL – here). AutoGPT is an agent implementation that has made specific decisions on overall architecture and prompting strategy.

    def generate_prompt_string(self) -> str:

        """Generate a prompt string.

        Returns:

            str: The generated prompt string.

        """

        formatted_response_format = json.dumps(self.response_format, indent=4)

        prompt_string = (

            f"Constraints:n{self._generate_numbered_list(self.constraints)}nn"

            f"Commands:n"

            f"{self._generate_numbered_list(self.commands, item_type='command')}nn"

            f"Resources:n{self._generate_numbered_list(self.resources)}nn"

            f"Performance Evaluation:n"

            f"{self._generate_numbered_list(self.performance_evaluation)}nn"

            f"You should only respond in JSON format as described below "

            f"nResponse Format: n{formatted_response_format} "

            f"nEnsure the response can be parsed by Python json.loads"

        )


        return prompt_string


// Prompting strategy in the LangChain AutoGPT implementation

An immediately achievable next step to making these agents useful in practice is expanding their action space: LangChain tools or AutoGPT plugins. These modules define the extended set of commands that an agent can perform. Examples include searching Google or writing some code on the fly. The open source community might expand this finite set of actions with, for instance, Twitter integrations to let the agent read and post tweets, or payments integrations to navigate checkout flows. This is where LangChain and AutoGPT excel: due to recent attention, developers flock to these projects to build plugins. It’s a really interesting moat where, similar to proof-of-work consensus in blockchains, developers are incentivized to build on the most complete plugin ecosystem—the longest chain, so to speak. The hard part is that the set of actions a user can take on the internet is near infinite. To get agents that can do everything humans can do online the long term solution is to have agents reliably write their own procedural code to gracefully handle novel cases, but we need a step function improvement in models before this future is within reach.


LANGCHAIN TOOLS

Currently, agents run like your run-of-the-mill MBA graduate or entry level consultant: they are very good at describing plausible solutions but very poor at executing on them. Put another way, the Act component of ReAct performs poorly in unconstrained environments (ReAct is constrained to a predefined action space), and it is clear that the agent isn’t able to reason at a deeper level about novel situations or problem solve on the fly. A concrete example of this is that agents often hit a wall when the output of an action isn’t what they expect: they try to pull up a tweet, hit a 403: Unauthorized error, and are not quite sure what to do next as they have no notion that calling Twitter might give a different response if the user is not logged in. Practically speaking, the current generation of agents still need quite a bit of human intervention and direction to be effective.

So what’s next?

Building a working autonomous agent in an unconstrained environment is an open research problem. We are still far from this reality. That said, models are improving at an accelerating clip. AutoGPT-like agents represent an interesting practical experiment on agents in an unconstrained environment. Through experimentation and iteration on architecture and prompting strategy and enough open source contributions in the action space (plugins), this approach could plausibly start to bridge the gap between AI doomer fantasy and reality. Here is my agent wishlist to try to get us there:

  • A better headless browser. The kinds of agent directives that work best are the ones that involve researching things on the internet. This works well when the agent starts with a Google search. It works less well when the agent then visits an arbitrary web page. Not only do agents need to understand and gracefully handle various HTTP errors, but also parse an arbitrarily large webpage and potentially execute Javascript to fully mimic a human user experience. Then, the agent would have to map the perceived UX to a limited context window. This is a very hard problem, but if you have a potential solution, here are some places to start.
  • Better command selection. Currently, chat completion is responsible for command selection (“determine which command to use…”). This leads to some pretty crazy hallucinations: for instance, the birthday planning agent shown below tries to reference as an argument a file that doesn’t actually exist. There is probably some clever combination of prompting and procedural code that can minimize hallucinations and help guide the agent in the right direction. For instance, you could try cosine similarity between the embedding vectors across more descriptive versions of current commands and the chat completion response. Or you could try to craft the prompt in a way to minimize edge cases or restrict the creativity of completion outputs. It’s unclear what the right solution is, but there is a lot of room for improvement here. As an aside, this a great opportunity to explore toolformers at the model level.

    {

      ...

      "speak": "Let's start by analyzing any previous conversations or notes we have about the daughter's interests and preferences for her birthday party. This will help us come up with creative and unique ideas for themes, decorations, activities, and entertainment that align with her incentives and preferences. Then we can move on to managing the logistics of the party, including venue selection, guest list management, catering, and budgeting. Finally, we should anticipate and address potential issues or challenges that may arise during the planning and execution of the party to minimize stress and maximize satisfaction."

    },

    "command": {

      "name": "read_file",

      "args": { "filename": "daughter_preferences.txt" }

    }


// Example hallucination (no such file exists!)

  • Optimizations around the action space. While the LangChain agent is parameterized by default (you decide which tools to pass on instantiation), AutoGPT defaults to a set of out of the box actions. Some of these actions give agents a ton of functionality: they can read, write and execute arbitrary code and files. In the best case, this will make agents truly autonomous and self-improving. However, agents are not yet able to generate code and files in productive ways more reliably than they are to fail or hallucinate in doing so. This leads agents to get stuck more often than it leads to novel and working outcomes. The suggestion here is threefold: 1) do a bit of “hyperparameter optimization” around which subset of actions lead to the best agents—and for LangChain, include this as the default set of tools; 2) allow agents to change the action set on the fly, either automatically once they have hit an edge case and run into a loop (should be easy to detect) or manually on user input; 3) build out guardrails, input validation, and better failure mode handling for existing actions (similar to the suggestion for a better headless browser). Also, for AutoGPT, potentially remove some of the more powerful “out there” commands until the models get better.
  • A discriminator to rank actions. The recent Generative Agents paper described a strategy that ranked observations in memory by some linear combination of “recency,” “importance,” and “relevance.” We could try implementing something like this for both memory context and task selection/action prioritization. The devil is in the details. For instance, the paper suggested a separate prompt to rank importance (“on a scale of 1 to 10…”); however, depending on the use case, “importance” could be better estimated by a numerical model (e.g. likelihood estimator) depending on the objective. There is likely no one size fits all solution, and the best approach is likely to start with a simple prompt and go from there.

RANKING ACTIONS FROM THE GENERATIVE AGENTS PAPER

  • More tools and plugins. Over the past two weeks, the number of approved plugins on AutoGPT has ballooned from 2 to almost 20—from Telegram to Wikipedia to crypto. LangChain tools are just as comprehensive. Each additional working plugin expands an agent’s action space and gets it that much closer to mimicking a human on the internet.
  • Agent-to-agent messaging. As agents are deployed in the wild, they can start communicating with each other in novel ways. These ideas are explored in multi-agent simulation environments, but AutoGPT and other unconstrained agents currently operate in single player mode. Messaging could introduce the possibility of more complex relationships between agents with different directives, such as student-coach or competitive peer-peer. This would require a persistence layer for agents, the ability for agents to query all other agents, and a messaging layer, potentially with a discriminator to rank messages by importance that are received between each iteration of the agent (h/t @nicktindle for this observation).
  • Better models. This is a must-have for the frequent gaps in logic the agent exhibits. We don’t quite get there with GPT-4—maybe the next generation of models? In addition, current autonomous agents are slow and expensive. Perhaps it is worth exploring fine-tuning on agent trajectories to make them faster, cheaper and effective for a subset of tasks. Toolformers remain underexplored but promising.

If there is one takeaway from all of this, agents’ reasoning ability is pretty good but their action-taking aspect is still pretty rudimentary. In research, agents largely run in constrained environments with limited abilities to act, like toddlers playing with toy cars in a sandbox. With autonomous commercial agents, we are seeing the first experimental attempts to have agents run in the wild unconstrained. We are giving these toddlers actual cars, so to speak. This is exciting and unprecedented, and would represent a zero-to-one improvement in agents once they work reliably. However, these toddlers don’t know how to drive and are currently crashing into most obstacles that come their way. Do we replace the cars with Tonkas or do we hope that the toddlers grow up? I have made some suggestions on the toy car side in the hopes of making some surprisingly competent toddlers, but if we want truly working autonomous agents, we also need to wait for better models.

Thanks to my colleagues at Sequoia, Lauren Reeder and Charlie Curnin, and to Ankush Gola from LangChain and Michael Graczyk for their thoughtful review and helpful suggestions for this post.