On Being a Scientist: A Guide to the Responsible and Ethical Conduct of Research: Fourth Edition 2026

Mike's Notes

For future reference. It can be read freely online.

Resources

References

  • National Academies of Sciences, Engineering, and Medicine. 2026. On Being a Scientist: A Guide to the Responsible and Ethical Conduct of Research: Fourth Edition. Washington, DC: The National Academies Press. https://doi.org/10.17226/29488. 2026. On Being a Scientist: A Guide to the Responsible and Ethical Conduct of Research: Fourth Edition. Washington, DC: The National Academies Press.

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

22/10/2026

On Being a Scientist: A Guide to the Responsible and Ethical Conduct of Research: Fourth Edition 2026

By: Megan Lowry
The National Academies of Sciences, Engineering, and Medicine: 17/09/2026

Megan Lowry: Megan Lowry is Manager, Media Relations, of the Office of News and Public Information at the National Academies of Sciences, Engineering, and Medicine.

...

WASHINGTON — The National Academies of Sciences, Engineering, and Medicine today released an update of On Being a Scientist: A Guide to the Responsible and Ethical Conduct of Research. This fourth edition is organized into online, thematic modules, and reflects the evolution of the scientific enterprise since the last version in 2009 by including new topics such as research security and responsible collaboration.

On Being a Scientist introduces students and early-career researchers across disciplines and sectors to the fundamental values that guide responsible research — such as objectivity, honesty, and openness — and explores the increasingly complex research contexts in which they need to be applied. The report was mandated by Congress.

“The research enterprise has changed immensely in recent years because of increasing collaboration, growing global competition, and emerging technologies like AI and CRISPR; these changes offer exciting opportunities for the advancement of science but also raise new ethical questions,” said Michele Masucci, vice chancellor for research and economic development for the University System of Maryland, and co-chair of the committee that developed the guide. “On Being a Scientist is meant to help students and early-career researchers navigate this evolving landscape. We hope the guide will also serve as a framework for responsible research practices for the entire scientific community.”

This guide’s release as online modules rather than as a printed book reflects how users typically access the guide and will make it easier to update going forward. The new edition follows previous printed versions published in 2009, 1995, and 1989.

While directed primarily toward undergraduate and graduate students, postdocs, and early-career researchers, the guide is also useful for faculty who teach research ethics and for established researchers who mentor people in research activities across range of settings — academic institutions, government agencies and laboratories, and industry.

“The dynamic, ever-shifting research environment makes the basic tenets of responsible science — including rigor, integrity, and transparency — even more critical to the effective functioning of the research enterprise and to maintaining public trust,” said National Academy of Sciences President Neil H. Shubin, who co-authored a preface to the new edition with Tsu-Jae Liu and Monica Bertagnolli, the current presidents of the National Academy of Engineering and National Academy of Medicine, along with the immediate past presidents of the NAS and NAM. “This guide is an essential resource to help set norms for ethical conduct and encourage practices that promote the use of science for societal benefit.”

The guide includes an introduction that offers an overview of responsible research and scientific values, together with the following four modules:

  • Collaborating Responsibly provides guidance on collaborating productively across disciplines and institutions, which is increasingly important as the scale and complexity of science grows. This module explores how to enter collaborations intentionally, structure them, manage challenges, and learn from experience.
  • Earning and Maintaining Trust focuses on the daily practices and professional commitments that establish trustworthiness; the institutional structures — including policies, cultures, and accountability measures — that support trustworthy science; and trustworthy communication with the public.
  • Openness and Security offers guidance on navigating the “productive tension” between openness about methods, materials, and data — which is important to evaluating findings and advancing science — and research security, which protects research from misuse, theft, or unintended harm.
  • Responsible Conduct and Stewardship of the Research Process covers the norms and practices of responsible science, such as the judicious use of resources, appropriate oversight and protections for research participants and subjects, prevention of mistakes and negligence, and adherence to norms of ethics and integrity.

“No single resource can fully cover the wide range of situations researchers encounter in their work,” said committee co-chair Alan Leshner, chief executive officer emeritus of the American Association for the Advancement of Science. “On Being a Scientist offers an introduction to these issues and ways to think about them. The goal is to help researchers recognize the relevant considerations when they arise, ask the right questions, and know where to look for further guidance.”

The project — undertaken by the Committee on On Being a Scientist: An Online and Updated Guide to the Responsible and Ethical Conduct of Research — was sponsored by the National Science Foundation, the U.S. Department of Energy, and the Alfred P. Sloan Foundation.

The National Academies of Sciences, Engineering, and Medicine are private, nonprofit institutions that provide independent, objective analysis and advice to the nation to solve complex problems and inform public policy decisions related to science, engineering, and medicine. They operate under an 1863 congressional charter to the National Academy of Sciences, signed by President Lincoln.

Contact:

Megan Lowry
Manager, Media Relations
Office of News and Public Information
202-334-2138; email news@nas.edu

CDC at WHOOP: Self-Service Replication for Hundreds of Postgres Tables

Mike's Notes

This article was mentioned in a recent Data Engineering Weekly from Ananth Packkildurai.

"WHOOP writes about replacing costly database replication that required data engineers to handle table and column changes. Its system records changes continuously, updates analytical copies every four hours, and automatically carries new tables and columns downstream. This reduces computing costs and lets service teams manage replication themselves, while teams needing fresher data can read the change records directly." - Ananth Packkildurai

Lots of great lessons here about implementing Change Data Capture (CDC).😎

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >Subscriptions > Data Engineering Weekly
  • Home > Handbook > 

Last Updated

21/10/2026

CDC at WHOOP: Self-Service Replication for Hundreds of Postgres Tables

By: Jack Leitch
WHOOP: 26/09/2026

Jack Leitch: Staff Software Engineer, Data Platform.

...

A software engineer at WHOOP adds a new column to a Postgres table on Monday morning. By that afternoon, the column is queryable in Snowflake and available to any downstream model or dashboard, alongside every other column on that table. Nobody filed a ticket. No data engineer touched anything.

Change Data Capture (CDC) is the pattern that makes this possible: every insert, update, and delete on a source table is streamed out as an event, and downstream systems consume that stream to stay in sync. That workflow is the point of the CDC platform we run today, and getting there meant deliberately avoiding the streaming-upsert pattern most CDC-to-lake pipelines converge on.

The system it replaced

Debezium captured row changes from Postgres and published them to Kafka as JSON, with no schema registry sitting in front of it, so downstream consumers were reading loosely typed payloads. A Spark Structured Streaming job then upserted those events into Iceberg tables on a thirty-minute trigger. It worked. It also had four problems that got worse as WHOOP grew:

  1. The upserts were expensive. We ran one Spark Structured Streaming job per table, so hundreds of long-running streaming jobs sat continuously reconciling their targets on a thirty-minute cadence. These tables were queried heavily and there was real value in keeping them fresh, but the compute cost of continuously running one streaming job per table was the dominant issue: we were paying a lot for freshness that most consumers did not strictly need.
  2. Schema changes were manual. A new Postgres column would not appear downstream until someone filed a ticket, and the fix usually meant backfilling the entire table. Every schema change turned into a coordination problem across teams.
  3. Observability was thin. We had Spark job metrics and Postgres-side replication slot lag, but nothing meaningful from Debezium itself. No per-connector throughput, no per-connector lag, no visibility into what the source connector was actually doing.
  4. Per-table configuration lived in the wrong place. Kafka partitioning, topic settings, and the list of replicated tables did live in the owning service's own repo, but they lived separately from the normal Postgres config and Liquibase migrations that engineers touched day to day. So they got forgotten. New tables would land in Postgres and quietly not make it into CDC, sometimes for months.

The pipeline itself was fine. Its operating model wasn't. Every design choice put Data Platform on the critical path for something a service team should have owned.

The core idea: split streaming from upserts

Most CDC-to-lake systems merge two concerns into one job. They take a stream of change events and, in real time, produce a materialized replica of the source table. That is a legitimate goal, but it forces continuous upserts, which are expensive to run at scale.

We split them apart.

  • A Bronze layer captures every CDC event exactly as Debezium emits it. It is append-only. Small files, cheap writes, no reconciliation.
  • A Silver layer materializes the actual Postgres replica. It reads Bronze and upserts into Iceberg, but only every four hours.

For consumers, this gives two clean options. If you need the latest state of a table and can tolerate a few hours of staleness, you query Silver, and the table looks exactly like the Postgres source. If you need lower latency or you specifically want change semantics (inserts, updates, deletes), you query Bronze and filter by the CDC timestamp.

Very few consumers actually need the latter, so continuous upserts were the wrong default.

End-to-end CDC architecture: Postgres to Debezium to Kafka to Flink to Bronze and Silver Iceberg tables

Debezium and Postgres publications

We run one Debezium connector per Postgres database. Each connector reads from a Postgres publication using logical replication, serializes each event as Avro against the AWS Glue Schema Registry, and publishes one Kafka topic per table using the convention postgres_cdc.<service>.<schema>.<table>.

The set of replicated tables is driven by the Postgres publication itself, and the publication is managed with Liquibase migrations that live inside the service team's own GitHub repository. Adding a table looks like this:

-- Liquibase migration in the service repo
ALTER PUBLICATION cdc_publication ADD TABLE new_feature_table;

Some services skip the per-table dance entirely and declare the publication as FOR ALL TABLES, so anything they add to Postgres is automatically replicated. New tables just appear.

That is the entire ceremony. Debezium picks up publication changes on the next poll. The Flink job downstream discovers the new topic within a minute and starts writing it to Bronze. Silver picks it up on its next run and backfills from the latest snapshot. No PRs on our side, no tickets. The service team owns the decision and the migration is code-reviewed in the same repo as the schema change that motivated it.

The other property that falls out of this design is automatic schema evolution. When Debezium sees a new column, it registers a new Avro schema in Glue. The Flink sink writes the new column to Bronze on the next commit. Silver sees the new field on its next run and adds it to the Iceberg schema. A single ALTER TABLE in Postgres propagates through the entire stack with no human involvement. Supported type changes work the same way: widening a decimal(10,2) to decimal(12,2), for example, flows through Avro's schema evolution rules and Iceberg's promotion rules without breaking any downstream reads.

Each connector deploys through an internal Terraform module that standardizes the operational plumbing: IAM, secrets, Datadog monitors, and the Glue registries. We also build our own Debezium container image, which bundles a small JMX metrics exporter alongside the upstream Debezium jars. That exporter surfaces connector-level metrics in Datadog: replication lag (MilliSecondsBehindSource), events seen, errored tasks, and restarts. Those metrics route to the owning team's oncall, not ours, which was a specific goal. If a service's connector is falling behind, the team that owns the database is the first to know.

Flink: one job per database, fan-in and fan-out

Bronze is written by Flink. Bronze tables follow the convention bronze.<service>.<schema>.<table>, so the Postgres source and its Iceberg replica are directly identifiable from each other's names.

There is one Flink job per Postgres database, and each job uses a fan-in / fan-out pattern: it subscribes to every Kafka topic under that database's prefix, then routes each record to the correct Iceberg table based on the topic name.

One Flink job per database subscribing to N Kafka topics and writing to N Bronze Iceberg tables

A naive design would run one Flink job per table, which means N deployments, N sets of checkpoints, and N connectors to Kafka. For a service with fifty tables, that is fifty jobs to autoscale, monitor, and upgrade. The fan-in/fan-out pattern collapses that to one job per database, regardless of how many tables live in it.

The routing itself is handled by the Dynamic Iceberg Sink introduced in Iceberg 1.10.0. Each record carries its Iceberg table name derived from the source Kafka topic, and the sink handles the rest, including creating new tables when they appear. We rediscover Kafka topics every 60 seconds, so new tables added to a Postgres publication start flowing to Bronze within a minute of the publication change. No job restart is required.

Each Bronze record is the row plus five CDC metadata columns:

Column Meaning
cdc_op c (create), u (update), or d (delete)
cdc_ts Source database event timestamp
cdc_processed_at When Flink processed the record
cdc_source_lsn Postgres WAL log sequence number
cdc_source_tx_id Postgres transaction ID

Bronze tables partition by day(cdc_ts). Compaction runs nightly against the last two days and sorts by cdc_ts DESC, which optimizes for the most common access pattern: "give me the recent changes to this table." Older partitions are already fine.

Jobs deploy through the Flink Kubernetes Operator in Application Mode on our EKS clusters. The operator manages lifecycle, autoscaling, and savepoint-based restarts. An auto-discovery runner scans Glue for new CDC topic prefixes and posts to Slack if it spots a new service without a matching Flink deployment. We deliberately do not fully automate that step today because a handful of services carry PHI and need to be routed to a separate Iceberg catalog. Deciding which catalog a new service belongs in is currently a human call, but it is the kind of thing we plan to automate, with service owners declaring their routing intent at onboarding time rather than pinging us for a decision.

Why Flink instead of the MSK Connect Iceberg sink? We prototyped the MSK Connect option early and it did not hold up at our scale. The connector was still relatively immature, and we hit ceilings on throughput, checkpointing behavior, and observability that were hard to work around. Flink was more operational work to run, but it gave us autoscaling based on backlog, better checkpointing semantics, richer Prometheus metrics, and the ability to add custom routing logic.

Silver: making upserts cheap

Silver is where the actual Postgres replicas live, and it is where the two-layer architecture pays for itself.

Every four hours, a Spark job runs per service. It does three things: figures out what to process, runs MERGE INTO for tables that already exist, and backfills tables that just appeared.

Discovery. The job queries the Glue Schema Registry to find every CDC-enabled table for the service and derives the primary key for each one from the Avro key schema. No manual PK configuration lives anywhere, though we do allow overrides for edge cases (for example, a table with a mutable primary key needs its logical merge key to be declared explicitly).

Backfill (new tables only). For a table that has no Silver counterpart, the job takes the latest automated Aurora snapshot, exports it to S3 in Parquet using Aurora's built-in snapshot export, and merges it into a fresh Silver table. There is one subtle guard here: if the snapshot predates the moment the table was first added to CDC (which we know because Glue records schema creation time), we defer the backfill until a newer snapshot is available. Otherwise we would be seeding from a snapshot that misses events already in Bronze, and the reconciliation from there onward would look correct but silently miss rows.

Incremental (existing tables). The job reads Bronze events since the last watermark, groups by primary key to pick the latest state per row, and executes MERGE INTO on the Silver table. Deletes propagate the same way (via cdc_op = 'd').

Tables get batched into groups of five per Spark job to balance parallelism against resource cost. Each service's Silver flow is independently scheduled.

Four layered optimizations inside the Silver Iceberg table are what let MERGE INTO run in minutes rather than hours. They turn each merge from a full-table shuffle into a targeted, well-pruned operation, and without them Silver would not be cost-viable at our scale.

Bucket partitioning on primary key

Silver tables partition by bucket(N, primary_key), where N is picked per table to target roughly consistent file sizes. Bucket partitioning distributes rows evenly across a fixed number of buckets regardless of PK distribution. That evenness is not just aesthetic. It is a prerequisite for the write-side optimization below.

Sort order and bloom filters within each bucket

Data within each bucket sorts by primary key, and bloom filter indexes are configured on the PK columns. Together they give Spark tight file-level pruning during MERGE: skip any file whose min/max range excludes the incoming PKs, and within a matching file, skip any Parquet row group whose bloom filter says the PK is definitely not present.

The read-side benefit for Snowflake is more limited. Snowflake's Iceberg reader uses min/max file statistics for pruning (which is why sorting matters) but doesn't currently use Parquet bloom filters. So the bloom filters are a Spark-side win; the sort order helps both.

Bucket pruning and Storage Partitioned Join

Two related optimizations do most of the write-side work. They're often talked about together but they are technically distinct.

The first is bucket pruning. When Spark plans a MERGE INTO between an incoming CDC micro-batch and the Silver table, it hashes the incoming primary keys, figures out which buckets they map to, and reads only those buckets from the target. No full table scan.

A five-key incoming batch hashing to buckets 3, 17, 42, 88, 101, with only those buckets highlighted in the 128-bucket Silver table

The second is Storage Partitioned Join. Because both sides of the merge share the same bucketing scheme, Spark can co-locate rows from the incoming batch with the rows already in the target and skip the shuffle that a normal join would need. Without SPJ, Spark would have to hash-repartition the entire target table to line it up with the incoming batch, which is often the largest single cost of a merge at scale.

Both optimizations depend on the same thing: an evenly bucketed target table. Skewed data breaks both, because a single hot bucket dominates the read on one side and the shuffle-avoidance on the other. That is why bucket partitioning is the foundation for everything else in this section.

SPJ plus bucket pruning are what made Silver viable. Merge jobs that would otherwise scan and reshuffle terabytes now touch a small fraction of the target table, and the runtime savings are order-of-magnitude.

Merge-on-Read

Silver tables use Iceberg's Merge-on-Read mode. Instead of rewriting entire Parquet files when a row updates, Iceberg writes a position delete file that marks which rows are stale. That skips the file rewrite on the write path, which is where most of the MERGE cost would otherwise land. Reads pay a small tax at query time to reconcile data and delete files, which we resolve during nightly compaction. Silver compaction sorts data back by primary key, which restores clean read performance and keeps future SPJ merges efficient.

We can afford these Silver optimizations because Silver runs every four hours rather than continuously. Larger batches amortize the fixed overhead of MERGE planning, and a job that runs six times a day is one we can actually tune, rather than firefight.

For teams that need lower latency, they read Bronze directly and filter on cdc_ts. Recent changes are cheap to scan there because Bronze compaction sorts by cdc_ts DESC and prunes small files aggressively for the last two days.

Audit trails from the same pipeline

CDC turned out to be a natural place to solve a problem that would otherwise need its own audit system.

Services that need audit trails on sensitive tables can opt into emitting audit context alongside their writes. When enabled, the service calls pg_logical_emit_message(true, 'audit', ...) inside the same transaction as the data change, embedding metadata like the acting user ID and the SQL query. The true argument matters: it makes the message transactional, so the message and the row change share the same xid and either commit together or not at all.

Debezium captures both. Flink writes row changes to their normal Bronze table and logical messages to a dedicated pg_logical_messages Iceberg table partitioned by day(cdc_ts) and the source table name. In Snowflake, a complete audit trail is a join:

select data.*, audit.user_id, audit.sql_query
from   postgres_cdc_prod.svc.sensitive_table as data
join   postgres_cdc_prod.svc.pg_logical_messages as audit
  on   data.cdc_source_tx_id = audit.cdc_source_tx_id
where  data.cdc_ts  >= dateadd(day, -30, current_timestamp())
  and  audit.cdc_ts >= dateadd(day, -30, current_timestamp());

Both sides get filtered by cdc_ts so partition pruning fires on the audit table as well as the data table. Postgres's transactional guarantees do the rest: every write to an audited table shares its transaction ID with any audit message emitted in the same commit, so the join gives you the transaction-level context (who, when, why) for any change. What the pipeline itself buys us: no separate audit system, and no risk of the audit trail drifting from the data.

The responsibility model this creates

The technical architecture reflects an organizational choice. Service teams own their data. Data Platform owns the infrastructure.

Component Owner
Debezium connector (via Terraform module) Service team
Publication contents (via Liquibase migration) Service team
Flink Bronze jobs Data Platform
Silver Spark jobs, backfills, maintenance Data Platform Data Platform
Snowflake catalog integration Data Platform Data Platform

Adding a table to CDC involves editing exactly one file, in the service's own repo. Everything downstream follows automatically. When a schema changes, no ticket is filed, because there is nothing for us to do. If we did our jobs right, Data Platform's involvement in the average CDC operation is zero.

For the analytical and modeling tables downstream of this (the ones that need explicit schema control and reproducibility), we manage them through Glacierbase, which handles the versioning problem for tables where controlled schema drift is not acceptable. The two systems complement each other: CDC handles ingestion where drift is expected and automatic; Glacierbase handles curated tables where every change should be a reviewed migration.

What this all buys

The two-layer split was about cost, not elegance. Continuous upserts were expensive, most consumers did not need the freshness they were paying for, and separating Bronze from Silver let us make the expensive operation infrequent and the cheap operation continuous. That in turn made the Silver optimizations worth investing in, because a job that runs six times a day can absorb serious tuning work.

Publication-driven self-service came out of the same accounting. Data Platform was the bottleneck for every schema change across every service, so we moved the source of truth into the service's own Liquibase migrations. New tables and new columns propagate downstream without our involvement.

The engineer from the opening paragraph doesn't need us. That is the whole design.

OfficeParser

Mike's Notes

OfficeParser is a great open-source tool by Harsk Ankur for office documents. I found this gem through Ray Camden's blog newsletter.

Harsk's blog is detailed and useful for understanding different file formats and how to convert between them.

Could be useful in the future for Pipi-generated docs.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library > Subscriptions > RaymondCamden.com
  • Home > Handbook > 

Last Updated

20/10/2026

OfficeParser

By: Raymond Camden
RaymondCamden.com: 17/10/2026

Raymond Camden: My name is Raymond Camden. I'm a married father of eight living in beautiful Lafayette, Louisiana. I'm a developer evangelist/advocate looking for my next role. Most of my time is spent writing, researching, or presenting. When I'm not behind a computer, I'm an avid Xbox/Playstation player, enjoy movies, and read like crazy.

...

I've been singing the praises of the officeParser library for a while. It's an open source JavaScript library that can parse, extract, and convert between PDFs and Office documents, as well as OpenOffice, ePub and more. My demos have all focused on client-side examples but obviously the library would work great in a server side workflow as well. I haven't worked at Adobe for years now, but I still get strangely excited working with documents. I'll just add that to the long list of reasons I'm a big nerd.

A few weeks ago, I reached out to Harsh Ankur about an idea inspired by Adobe's Document Generation API. This was probably my favorite API in their suite of PDF services. You take a Word document, put in tokens, and call the API with your data. The API would swap out the tokens with real data and give you a PDF in return. (Or Word again if you wanted that.) I suggested this to Harsh and then - voila - he added it!

At this time, the token system is simply a replacement system. You add My name is in your Word document, and if you pass { "name":"Juniper Montage"} via the library's APIs you'll get My name is Juniper Montage on the way out.

This new support is a core API, OfficeTemplate, and supports both replacing values from a regular JavaScript object or returning an array of results from a JavaScript array of objects. (Which, by the way, is something I was trying to get Adobe to add to their APIs. One of their competitors, DocuGenerate, has had this for a while.)

Here's how simple it is to use. First, create and save a Word document. Formatting doesn't matter - make it as pretty or simple as you would like. Here's an example with a few tokens:

Customer Report for {{

customerName}}

On {{ date }}, the customer {{ customerName }} currently has {{ productsTotal

}} in the market. Their gross profit for the year {{ reportYear }} was {{ profit }}

with losses of {{ losses }}.

Our recommendation: {{ recommendation }}


Report created by {{ reporter }}.

Imagine this is stored as customerReport.docx. To turn this into a PDF, you would first import the relevant libraries (in the sample below I'm using an npm installed version, not CDN):

import { OfficeConverter, OfficeTemplate } from 'officeparser/browser-native-pdf'

Then - grab the Word document as binary data:

const response = await fetch('/assets/customerReport.docx')
const template = new Uint8Array(await response.arrayBuffer())

Get some sample data:

const data = {
  customerName: 'Northwind & Co',
  date: '2026-10-16',
  productsTotal: 7,
  reportYear: 2026,
  profit: 2500,
  losses: 125,
  recommendation: 'Continue growing. Duh',
  reporter: 'Juniper Montage',
}

Next, use the library to convert the template:

const docx = await OfficeTemplate.render(template, { data })

Finally, turn the rendered document into a PDF. The result's value is the PDF's binary data:

const { value: pdfBytes } = await OfficeConverter.convert(docx, 'pdf', {
generatorConfig: { pdfConfig: { engine: 'native' } },
})

At this point you can do whatever, like open it in a new tab:

const pdf = new Blob([pdfBytes], { type: 'application/pdf' })
window.open(URL.createObjectURL(pdf), '_blank')

As a note, this requires an additional package, only in the browser, pdf-lib.

A Real Demo

To show this in action, I built a simple demo that lets you fill in a form, and get the result as a PDF.

The entry form

This form is tied to a bit of JavaScript that grabs the values, makes use of Intl to format dates, numbers, and currencies, and then uses officeParser to create a dynamic PDF from your values. Here's an example:

Customer Report for Foo Bar Products

On October 22, 2026, the customer Foo Bar Products currently has 390,189 in the market. Their gross profit for the year 2026 was $9,100,000.00 with losses of $300,000.00.

Our recommendation: They should keep working on their stuff. And things.

Report created by Juniper Montage.

Try this yourself here:

You can find the complete source for this here:

As I mentioned above, currently the token system is relatively simple, but I've been talking with Harsh about some improvements that I bet will land pretty soon. Let me know what you think!

How devs are using this viral GitHub repo that reverse engineers anything

Mike's Notes

"A GitHub repo called REA, short for Reverse Engineer Anything, lets your agent look inside any app and build your own version." - The Code

It finally happened. These open-source apps can be freely downloaded and also hosted on a website using WebAssembly. All built in Rust.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library > Subscriptions > The Code
  • Home > Handbook > 

Last Updated

19/10/2026

How devs are using this viral GitHub repo that reverse engineers anything

By: Editor
The Code: 17/10/2026

...

By now, you’ve probably seen that one dev rebuilt Adobe’s software and open sourced all of it. That trend isn’t going to stop. Now a GitHub repo called REA, short for Reverse Engineer Anything, lets your agent look inside any app and build your own version.

In this issue, you’ll find 3 workflows devs are using to reverse engineer software, plus more fun projects to learn from. Let’s get started:

Todays Brief

  • What is REA or Reverse Engineer Anything
  • Workflow #1: Mod a game with Claude Code
  • Workflow #2: Rebuilt a feature from another app
  • Workflow #3: Build better software for your hardware

What it is. REA is an MCP server that lets you rebuild almost any software out there. It decompiles whatever app you point it at, then guides your AI agent to recreate the exact features you want.  feature work? Can you build a version of it for me?" Your agent passes that request to REA, which connects it into decompiler (the tools engineers use to turn compiled apps back into human-readable code). 

With each pass, REA returns pseudocode showing what the code does, along with supporting evidence and anything it couldn't solve. Your agent keeps iterating until the feature is completely rebuilt. 

To get started, have your agent install REA for you, or run "npx rea-agents setup" yourself and restart your agent. Just make sure to stick to open-source software and apps or devices you own, since many terms of service forbid reverse engineering. 

Here are 3 workflows worth trying out this weekend:

... [See The Code for more details]

Graduate Student Proves a Quantum Uncertainty Principle for Fractals

Mike's Notes

This Quanta Magazine article about Alex Cohen's discovery is also available as a podcast. His work builds on the work of many who came before him.

The discovery connects to porosity (percolation?), the Uncertainty Principle via Fourier Transform, etc., and might be very useful to add to Pipi. I need to investigate it further.

Resources

References

  • Fractal uncertainty in higher dimensions v2, by Alex Cohen, 5 Oct 2024. arxiv.org/abs/2305.05022.

Repository

  • Home > Ajabbi Research > Library > Subscriptions > Quanta Magazine
  • Home > Handbook > 

Last Updated

18/10/2026

Graduate Student Proves a Quantum Uncertainty Principle for Fractals

By: Shalma Wegsman
Quanta Magazine: 12/08/2026

Shalma Wegsman: Staff Writer.

...

The math, which combines chaos, quantum theory, and infinitely complex fractal structures, has been called a “foundational result.”

Introduction

At the quantum scale, tiny particles behave in bizarre ways. One reason for this is the uncertainty principle, which says that the more you know about where a quantum particle is, the less you can know about how fast it’s moving, and vice versa. Recently, this rule got a rare upgrade.

The new uncertainty principle relates to fractals, shapes that remain equally complex no matter how much you zoom in on them.

Around a decade ago, Semyon Dyatlov, a mathematician at the Massachusetts Institute of Technology, was studying whether quantum particles behave differently than ordinary particles when put into the same chaotic situations. Sometimes, an object moving chaotically can become trapped into following a fractal-like path forever. Could quantum particles do the same?

The Quanta Podcast

The uncertainty principle tells us that the more you know about where a quantum particle is, the less you can know about how fast it’s moving, and vice versa. Recently, this rule got a rare upgrade.

Quantum particles tend to spread out like waves, which blurs their exact location. To figure out whether quantum particles blur too much to take on these intricate trapped paths, Dyatlov needed a new uncertainty principle — one that could tackle fractals.

In 2016, with key ideas from Jean Bourgain — a renowned mathematician who died shortly after this work — Dyatlov proved the fractal uncertainty principle for one-dimensional fractals, which look like jagged lines. These lines can represent the paths taken by objects moving in two dimensions, like balls traveling around a billiard table. That fall, Dyatlov and Bourgain gathered mathematicians from around the world in New Jersey for a workshop, hoping to extend the proof to higher dimensions. An extended proof could be used to study the three-dimensional world and would become a universal mathematical tool in its own right.

But the task proved too difficult. By the end of the workshop, “nobody really believed that it could be done,” said one of the attendees, Frédéric Naud, a mathematician from Sorbonne University.


Alex Cohen’s proof of the fractal uncertainty principle has already proven influential across math.

Hertz Foundation

It wasn’t until years later that Alex Cohen, while a doctoral student at MIT, finally made a breakthrough. In a paper published in 2025 in the Annals of Mathematics, widely considered to be the field’s top journal, he extended the fractal uncertainty principle to all higher dimensions. The result became Cohen’s thesis and earned him an assistant professorship at New York University at the age of 25.

The fractal uncertainty principle is “a foundational result,” said Peter Sarnak of the Institute for Advanced Study — “a pretty remarkable achievement for a guy in his thesis.”

Already, this principle has revealed a new deep way that quantum particles differ from classical ones.

Pinball Wizard

Though the uncertainty principle may seem strange in the context of particles, it can also crop up in less mysterious forms. The briefer a sound, the less sure you can be about the tones that make it up. A short radar pulse can accurately locate a submarine, but it takes a longer signal to determine where it’s moving.

All these uncertainty principles, including the quantum one, arise from the same mathematical source. This deeper mathematical uncertainty principle applies broadly to any function — or any curve, roughly speaking, no matter how bumpy and wild it looks. It comes from an equation invented in the 19th century called the Fourier transform. Named for the Frenchman Joseph Fourier, the Fourier transform decomposes any function into a set of simple waves, each with a different frequency, or tone. Add those simple waves together, and you’ll get back your original function.

Mark Belan/Quanta Magazine

The uncertainty principle comes built in. A simple sine wave, which spreads infinitely throughout space, has one definite frequency, so its Fourier transform is a single peak. But the snap of a snare drum looks like a pulse as it travels through the air. Its Fourier transform has a wide spread of frequencies — a sound with no discernible pitch.

In general, the narrower a function, the more spread out its Fourier transform must be, and vice versa. Or, in other words, the more certain you are about where a function peaks in space, the less certain you can be about its frequency, or how it’s changing in time.

In quantum mechanics, a particle is described mathematically as a wave, with peaks in places where it’s most likely to be found. The Fourier transform of that wave describes the particle’s motion. This is why a quantum particle’s position and momentum can’t both be precisely known at once: They are related by a Fourier transform.

But what if you take the Fourier transform of a function that looks like a fractal?

To make sense of the question, it helps to imagine a simple kind of fractal: Start with a line and cut it into three segments. Then delete the center segment. Repeat the process: Cut each remaining segment into three, remove the center, and so on. The result is a fractal set called the Cantor set. The set has the feature of being what mathematicians call porous — it’s full of holes, like a sponge, at every scale.

Now imagine that this Cantor set lives on the x-axis, and at each point on this set is a peak representing a frequency.

The fractal uncertainty principle says that if you add together waves of that fractal-like set of frequencies, the resulting curve cannot also look like a fractal — it cannot be porous. The reverse also holds: If you take the Fourier transform of a fractal-like function, the result cannot be a fractal-like set of frequencies.

Such fractal-like functions might sound hard to come by. But they pop up when mathematicians study what happens to quantum particles — or waves more generally — in chaotic situations.

Like a ball bouncing around a pinball machine, an object experiencing chaos will often travel erratically around the entire space. “If you look at your path, it’s going to look like a random scribble,” said Elena Kim, a mathematician at Harvard University.

But in rare instances, an object can find stability in the chaos. In a flat pinball machine, a ball could stay trapped bouncing between three bumpers forever. The ball wouldn’t repeat the exact same path, but it would stay confined between the bumpers, bouncing off a slightly different spot each time. And if you marked each spot where the ball hits each bumper, you’d find that the marks make up something like the Cantor set.

This collection of marks is also called fractal dust. Most balls that hit the bumpers will fly off. “This is the dust that’s left,” said Maciej Zworski of the University of California, Berkeley.


Elena Kim, a mathematician at Harvard University, has used the fractal uncertainty principle to understand chaotic systems in unconventional spaces.

Steph Stevens

Unlike a pinball, however, a wave cannot be confined to a fractal-like path, according to the fractal uncertainty principle. If you tried to trap a wave between three bumpers, it would leak out and escape.

“So there’s something different about quantum and classical [chaos],” Dyatlov said. “And one ingredient that you can try to use for that would be uncertainty principles.”

Unfinished Knowledge

Cohen arrived at MIT in 2021 a self-described “young, energetic harmonic analyst” — harmonic analysis being a field of mathematics dedicated to studying functions and their Fourier transforms. As a doctoral student, he sat in Dyatlov’s talks about the fractal uncertainty principle. “He would end every talk being like, ‘We don’t have a higher-dimensional fractal uncertainty principle. I wish we had that!’” Cohen said. “I was like, OK, this seems like a fun thing to work on.”

His enthusiasm was soon checked. For one thing, mathematicians already knew of many situations where the fractal uncertainty principle would fail in higher dimensions. Cohen’s first challenge was to come up with a clean way to avoid these cases.

The typical requirement of being a fractal is to be porous — meaning that there are holes everywhere you look. The two-dimensional fractal called the Sierpiński carpet is an example; it’s built by dividing a square into a grid of smaller squares and removing the middle square, repeatedly. But while this shape has many holes, it’s also possible to draw a straight line that is fully contained within it.

Lines like these spell trouble for the fractal uncertainty principle. The Fourier transform of a vertical line returns a horizontal line, and vice versa. In two or more dimensions, both of these lines count as fractals — that’s because a line takes up no area and leaves most of the surrounding space empty, which satisfies the condition of having many holes. So any fractal that contains uninterrupted lines breaks the fractal uncertainty principle.

Cohen needed a new, more stringent kind of porosity. He came up with what he called “line porosity” — any line you draw on the fractal should encounter many holes. His proof excludes any fractals that don’t have this condition, including the typical Sierpiński carpet, but a modified Sierpiński carpet with much more empty space satisfies the rule.

With his assumptions in place, Cohen moved on to the actual proof. He quickly realized that this was unlike any Fourier-related problem he had worked on before. “I tried using all my tools to prove the fractal uncertainty principle, and none of them even came remotely close to working,” he said.

Feeling stuck, he went back to Dyatlov and Bourgain’s proof of the principle in one dimension and sought to understand exactly how it worked.

Dyatlov and Bourgain used an uncommon method in their proof. It involved isolating one peak from a fractal-like function at a time and showing that the Fourier transform of that peak would spread out. Doing this for all peaks, and considering how the Fourier transforms would add together, they proved that the total Fourier transform could never equal zero often enough to form a fractal — there wouldn’t be enough holes.


Jean Bourgain, seen here in 2012, was an author on more than 500 papers spanning much of mathematics.

George M. Bergman

Isolating each peak required constructing a very specific function that, when multiplied by the original fractal-like function, would pull out just the peak and be close to zero everywhere else. This is called a damping function, and it needs to be perfectly tailor-made to work. “This is a challenging thing to construct,” Cohen said. But he knew that if he could do it in higher dimensions, he could unlock the entire proof.

Cohen consulted Dyatlov about his plan to construct this special function. Before Bourgain died in late 2018, he too struggled with this problem, and he shared his unpublished notes with Dyatlov. Now, Dyatlov shared them with Cohen. “Bourgain was a legendary analyst,” Cohen said. Reading the note felt like “receiving this unfinished knowledge from him.”

The notes contained exactly the hint Cohen needed. “It just blew my mind,” Cohen said. “It really unlocked the problem for me.”

Before reading Bourgain’s note, Cohen had a few ideas for how to construct the damping function, but they were highly complicated and precise, like the designs for building a house brick by brick. The note revealed an unexpected way to do it. It involved taking a detour into complex analysis — the study of functions of imaginary numbers, which include the square root of negative 1. This detour allowed Cohen to build a much more flexible object, which could then be used to construct the damping function indirectly.

Semyon Dyatlov, a mathematician at the Massachusetts Institute of Technology, proved the one-dimensional fractal uncertainty principle in 2016.

Xuwen Zhu

Armed with this insight, Cohen then needed to find a way to create just the right version of this flexible object to produce a proper damping function. “To construct something like this that has very specific properties is highly nontrivial. It’s delicate,” said Wilhelm Schlag(opens a new tab) of Yale University, with whom Cohen studied as an undergraduate. “In two dimensions, nobody knew how to do that, and Alex came up with a brilliant construction of such a thing.”

Cohen stunned the math world when he posted the proof online in May 2023.

“His paper is very beautiful, and it made a huge impression,” Schlag said.

Later, Cohen found out that the trick revealed to him in Bourgain’s note wasn’t actually a secret. The method came from a well-known theorem from the 1960s called the Beurling-Malliavin theorem. “I thought that I had this special inside knowledge,” Cohen said. “I found out later that everyone in the field already knew about this strategy.”

Had he known that his insider tip was no secret, Cohen might have given up too soon. “I think I had a lot of confidence because I didn’t know other people had tried it,” he said.

Funhouse Chaos

Soon after Cohen shared his result, other mathematicians started using it to unlock new proofs about how waves behave in chaotic situations.

In nature, chaos appears in systems like turbulent water and the weather — situations where objects that start close together quickly end up in drastically different places. These systems are too complex to describe mathematically. Instead, mathematicians seeking to study chaos often turn to an odd kind of space that has chaos built in, called hyperbolic space.

In hyperbolic space, parallel lines diverge dramatically, getting farther from each other as you follow their paths. (It’s the opposite of a sphere, where parallel lines converge.) This means that small separations between objects can become huge down the line — the telltale sign of chaos.

“Of course, this looks nothing like predicting weather,” Dyatlov said. But hyperbolic spaces offer mathematicians a simple chaotic system to explore. “We study what we can handle, and we isolate phenomena. That’s a common thing to do in math.”

In 2017, Dyatlov and Long Jin from Tsinghua University in Beijing used the one-dimensional fractal uncertainty principle to prove that you can never trap a wave on a hyperbolic surface; it will always spread out until it touches every corner.

To do so, they imagined a region on the surface that a wave never enters, even after having infinite time to spread out. When they removed all the trajectories that entered that region, what remained was the same sort of fractal dust that appeared in the pinball example. Since the fractal uncertainty principle forbids a wave from being trapped on a fractal, no such region can exist — the wave must spread everywhere.

In 2025, Kim, along with Nicholas Miller(opens a new tab) of the University of Oklahoma, used Cohen’s higher-dimensional fractal uncertainty principle to extend the results to hyperbolic spaces of higher dimensions. “That’s the most spectacular application of the higher-dimensional one so far,” Sarnak said.

Many mathematicians are hoping to prove that waves experiencing chaos not only spread out completely but also spread out exactly evenly over the entire space. This is the statement of a famous 1994 conjecture by Sarnak and Zeév Rudnick, who were interested in the problem’s connections to number theory. “This is a huge open conjecture,” Kim said. “It’s a really difficult problem that a lot of people care about.”

Proving this conjecture would show that waves experiencing chaos move in ways that look surprising and random at small scales but are simple at the macroscopic level.

“It’s like those Monet paintings,” Dyatlov said. “When you go very close, they have a lot of microscopic features. There are a lot of brushstrokes.” But if you stand back and squint, “it just looks uniformly colored.” This is different from classical chaos, where objects can get stuck in complicated and detailed paths.

The impact of the fractal uncertainty principle won’t stop with quantum chaos. The tools of Fourier analysis are used in many fields of mathematics and form the backbone of any industry that relies on signal processing. “It’s a foundational fact about Fourier analysis,” Sarnak said. “We haven’t seen all the applications yet.”