Showing posts with label Research. Show all posts
Showing posts with label Research. Show all posts

Using Google Blogger API v3

Mike's Notes

On a Sandy Beach is a publication of Ajabbi Research.

Changes needed

Enable other people at Ajabbi Research to also contribute to On a Sandy Beach using the Workspaces for Research UI.

It looks very straightforward. This job will be done once the Workspaces for Research are available for researchers to use.

Preparation

Last year, all existing blog posts were reformatted.

New setup

The Pipi CMS Engine (cms) will format, store in a database, and export content to On a Sandy Beach, hosted on Google Blogger, using the Blogger API ver 3.

Either XML/Atom or JSON can be used.

Steps

  1. Import the existing blog posts into the CMS Engine via the Blogger API
  2. Reformat every post (see 4th resource link below)
  3. Store in the CMS database
  4. Render posts
  5. Export back to Blogger via the Blogger API.
  6. All future posts will be created by a human using a workspace, transferred to the CMS Engine, processed, and then published to Google Blogger.

Notes

Here are some initial notes taken from the Google Blogger API documentation and other sites, tweaked using Gemini and rewritten by Grammarly.

These notes will be presented using slides at tomorrow night's online Open Research Group meeting.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

09/02/2026

Using Google Blogger API v3

By: Mike Peters
On a Sandy Beach: 07/02/2026

Mike is the inventor and architect of Pipi and the founder of Ajabbi.

Blogger supports manual import and export of blog content via the Blogger dashboard or the Blogger API. The export file format for both methods is Atom, an XML-based format that includes all posts and comments. 

Blogger API for Import/Export

Developers can use the Blogger API (v3 is the current version) to programmatically manage blog content. The export file generated via the manual process uses the same Atom format as the API's feed requests. 

Key Points for API Use

  • Authentication: All operations for private data (including import/export) require authentication, typically using OAuth 2.0.
  • Format: The API uses REST APIs and JSON for standard operations. The specific import/export functions rely on the raw XML/Atom format described in the developer guides.
  • Functionality: The API allows retrieving, creating, updating, and deleting posts, comments, and pages, enabling the creation of custom import/export tools. For example, you can use API clients for Python, Java, or Node.js to manage content programmatically.
  • Custom Import Scripts: Developers can write scripts (e.g., in PHP or Python) to import content from other sources into Blogger via the API, which involves parsing the source data and making POST requests to Blogger API endpoints. 

For detailed API documentation, refer to the Blogger API Developer site.

REST

The Blogger API is a RESTful interface provided by Google that allows developers to integrate Blogger content and functionality into their own applications. It enables programmatic access to resources such as blogs, posts, comments, pages, and users via HTTP requests and JSON. 

Key Features and Concepts

  • Resources: The API revolves around five core resource types:
    • Blogs: The central container for all content and metadata.
    • Posts: The primary published content items, meant to be timely.
    • Comments: User reactions to specific posts.
    • Pages: Static content (e.g., about me, contact info).
    • Users: Represents a non-anonymous person interacting with Blogger, as an author, admin, or reader.
  • Operations: Developers can perform various operations on these resources, including list, get, insert (create), update, patch, and delete.
  • Authentication:
    • Public Data: Requests for public data (e.g., retrieving a public blog post) only require an API key.
    • Private Data: Operations involving private user data (e.g., creating a post, editing a comment) must be authorised using OAuth 2.0 tokens.
  • API Version: The latest recommended version is Blogger API v3. Support for the older v2.0 API ended on September 30, 2024, so applications must be updated to continue functioning. 

How to Access the API

Google recommends using their client libraries, which handle much of the authorisation and request/response processing for you. Libraries are available for a variety of programming languages: 

  • Go
  • Java (get started with the Java client library)
  • JavaScript
  • .NET (install the Google APIs NuGet package)
  • Node.js
  • PHP
  • Python (install the client library using pip install google-api-python-client)
  • Ruby 

Alternatively, you can interact with the API directly using RESTful HTTP requests. The Google APIs Explorer tool lets you test API calls in your browser. 

  • Pipi API Engine (api)  using CFML
  • BoxLang API using BX

For full documentation and developer guides, visit the official Blogger API documentation site on Google for Developers. 

Older Data API (v1/v2) Format: Atom XML 

The previous versions of the Blogger API relied heavily on the Atom Publishing Protocol (AtomPub) and Google Data API feeds for managing blog content. The API used standard HTTP methods (GET, POST, PUT, DELETE) to transport Atom-formatted XML payloads. 

Important: Support for the v2.0 Google Data API ended on September 30th, 2024, so applications must use the latest version to continue functioning. 

Exporting and Accessing Feeds via XML

Blogger still uses Atom XML for blog syndication and content backup: 

  • Public Blog Feed: Every Blogger blog has a public Atom feed, typically at an address like yourblogname.blogspot.com/feeds/posts/default or://yourblogname.blogspot.com.
  • Content Backup: Users can back up their blog's posts and comments as a single .xml file from the Blogger dashboard.

    • Sign in to Blogger.
    • Select your blog.
    • Go to Settings > Manage Blog.
    • Click Back up content and Download XML file. 
    • This downloaded file is in a specific Atom format for import and export. 

REST in the Blogger API

The supported Blogger operations map directly to REST HTTP verbs, as described in Blogger API operations.

The specific format for Blogger API URIs are:

  • https://www.googleapis.com/blogger/v3/users/userId
  • https://www.googleapis.com/blogger/v3/users/self
  • https://www.googleapis.com/blogger/v3/users/userId/blogs
  • https://www.googleapis.com/blogger/v3/users/self/blogs
  • https://www.googleapis.com/blogger/v3/blogs/blogId
  • https://www.googleapis.com/blogger/v3/blogs/byurl
  • https://www.googleapis.com/blogger/v3/blogs/blogId/posts
  • https://www.googleapis.com/blogger/v3/blogs/blogId/posts/bypath
  • https://www.googleapis.com/blogger/v3/blogs/blogId/posts/search
  • https://www.googleapis.com/blogger/v3/blogs/blogId/posts/postId
  • https://www.googleapis.com/blogger/v3/blogs/blogId/posts/postId/comments
  • https://www.googleapis.com/blogger/v3/blogs/blogId/posts/postId/comments/commentId
  • https://www.googleapis.com/blogger/v3/blogs/blogId/pages
  • https://www.googleapis.com/blogger/v3/blogs/blogId/pages/pageId

The full explanation of the URIs used and the results for each supported operation in the API is summarised in the Blogger API Reference document.

Examples

List the blogs that the authenticated user has access rights to:

  • GET https://www.googleapis.com/blogger/v3/users/self/blogs?key=YOUR-API-KEY

Get the posts on the code.blogger.com blog, which has blog ID 3213900:

  • GET https://www.googleapis.com/blogger/v3/blogs/3213900?key=YOUR-API-KEY

REST from JavaScript

You can invoke the Blogger API from JavaScript using the callback query parameter and a callback function. When the browser loads the script, the callback function is executed, and the response is passed to it. This approach allows you to write rich applications that display Blogger data without requiring server-side code.

The following example retrieves a post from the code.blogger.com blog, after you replace YOUR-API-KEY with your API key.

<html>
  <head>
    <title>Blogger API Example</title>
  </head>
  <body>
    <div id="content"></div>
    <script>
      function handleResponse(response) {
        document.getElementById("content").innerHTML += "<h1>" + response.title + "</h1>" + response.content;
      }
    </script>
    <script     src="https://www.googleapis.com/blogger/v3/blogs/3213900/posts/8398240586497962757?callback=handleResponse&key=YOUR-API-KEY"></script>
  </body>
</html>

Data format

JSON (JavaScript Object Notation) is a common, language-independent data format that provides a simple text representation of arbitrary data structures. For more information, see json.org.

Blogger API operations

You can invoke a number of different methods on collections and resources in the Blogger API, as described in the following table.

Operation Description REST HTTP mappings
list Lists all resources within a collection. GET on a collection URI.
get Gets a specific resource. GET on a resource URI.
getByUrl Gets a resource, looking it up by URL. GET with the URL passed in as a parameter.
getByPath Gets a resource by looking it up by its path. GET with the Path passed in as a parameter.
listByUser Lists resources owned by a User. GET on a user owned collection.
search Search for resources, based on a query parameter. GET on a Search URL, with the query passed in as a parameter.
insert Create a resource in a collection. POST on a collection URI.
delete Deletes a resource. DELETE on a resource URI.
patch Update a resource, using Patch semantics. PATCH on a resource URI.
update Update a resource. PUT on a resource URI.

The table below shows which methods are supported by each resource type. All list and get operations on private blogs require authentication.

Resource Type Supported Methods
list get getByUrl getByPath listByUser search insert delete patch update
Blogs N Y Y N Y N N N N N
Posts Y Y N Y N Y Y Y Y Y
Comments Y Y N N N N N N N N
Pages Y Y N N N N N N N N
Users N Y N N N N N N N N
By , No comments:
Tags , , , , ,

How Science Actually Uses Machine Learning in 2025

Mike's Notes

A great summary of ML resources other than Generative AI. It was published in the Turing Post newsletter. The Turing Post is worth subscribing to.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library > Subscriptions > Turing Post
  • Home > Handbook > 

Last Updated

213/01/2026

How Science Actually Uses Machine Learning in 2025

By: Marktechpost
Turing Post: 18/12/2025

In this guest post, Marktechpost shares findings from the ML Global Impact Report 2025, an interactive research landscape now available at airesearchtrends.com. Based on an analysis of 5,000+ papers from Nature journals, the report shows that while the world obsesses over generative AI, classical methods still dominate real scientific research.

In fact, 77% of machine learning applications in science rely on traditional techniques like Random Forest, XGBoost, and CatBoost, not transformers or diffusion models. The gap between AI headlines and what actually runs in labs is much larger than most people realize.

State of current media

If you open any tech newsletter or media house, you’ll be convinced that science runs on GPT-4, diffusion models, and whatever architecture dropped last week.

What we found looks very different.

When we analyzed over 5,000 scientific articles published in Nature family journals between January and September 2025, a different picture emerged entirely.

Classical ML methods, like Random Forest, support vector machines (SVMs), and Scikit-learn workflows, account for 47% of all ML use cases in scientific research.

Moreover, adding established ensemble methods like XGBoost, LightGBM, and CatBoost, and traditional approaches, represents a dominant 77% of how scientists actually deploy ML models today.

But for neural networks and deep learning, it’s roughly 23%.

This has nothing to do with slow adoption but rather how research teams put forward their research.

In practice, researchers prioritize methods they can clearly justify during peer review. Novelty matters, but only after reliability, interpretability, and dataset sanity checks are satisfied. A Random Forest that reliably predicts protein binding affinity beats a transformer that might work better but introduces unpredictable failure modes.

When a paper depends on reproducibility and collaborators need to understand every step, novelty quickly becomes a liability. And in that context, reliability matters more than novelty.

For practitioners, the risk is obvious. Optimizing your stack around what trends on AI Twitter actively pulls you away from the methods that dominate real scientific workflows.

But who creates the tools versus who uses them?

The most consequential imbalance in ML research is who builds the tools versus who publishes with them. Nearly 90% of open-source ML tools referenced in 2025 scientific literature originate from the United States, where the foundational frameworks are powering imaging, genomics, and environmental science worldwide.

Yet the heaviest users aren’t American. China accounts for 43% of all ML-enabled papers in our dataset, compared to just 18% from the United States.

India holds third place and is climbing fast.

This imbalance shapes who sets research agendas and who executes them. Essentially, American institutions and companies build the infrastructure, while Chinese researchers publish the papers.

That said, non-US contributions to the tooling ecosystem are significant and growing. For instance:

  • Scikit-learn emerged from France.
  • U-Net, the workhorse of medical image segmentation, came from Germany.
  • CatBoost was developed in Russia by Yandex.
  • Canada produced foundational GAN and RNN architectures.

The US still dominates this layer of the stack, but it is not the only contributor.

What does this mean for the AI research community? The countries that define the frameworks shape how problems get formulated. When you inherit someone else’s tools, you often inherit their assumptions about what problems matter and how they should be solved.

Applied sciences and health research lead adoption

ML hasn’t penetrated all scientific fields equally. The heaviest adoption appears in applied sciences and health research, which are disciplines where prediction, classification, and pattern recognition map cleanly onto existing workflows.

Consider the specific applications we observed across thousands of papers:

  • In medical sciences, ML enables early-detection imaging, precision diagnostics, genomic sequence mapping, and mutation tracking. Cancer screening pipelines now routinely incorporate ensemble classifiers. Biomarker discovery increasingly depends on feature extraction from high-dimensional datasets.
  • In physical and materials sciences, researchers apply ML to advanced robotics, materials engineering, and complex physical simulations. Nanotechnology labs use neural networks to predict material properties before synthesis, saving months of experimental iteration.
  • In environmental sciences, large-scale Earth-observation analytics rely on ML for everything from monsoon prediction to disaster preparedness. Climate models incorporate machine learning components for pattern recognition in satellite imagery. Remote sensing has been transformed.
  • In agriculture, crop-yield forecasting and food systems resilience modeling now depend on ML pipelines trained on decades of historical data.

In most papers, ML appears as one step in an experimental pipeline, not the object of study.

In most domains, ML is not the focus of the paper itself. Instead, researchers publish papers about cancer or climate or crop yields that happen to use machine learning as methodology, showcasing that ML has transitioned from the research subject to the research infrastructure used in downstream problem-solving.

Collaborative research is predominant!

Contributions from the research community and our analysis make it clear that the lone genius scientist is a myth.

Most ML-enabled papers in our dataset include 2-15 institutional affiliations. Major studies rarely originate from a single organization.

But collaboration patterns differ dramatically by region.

To put this into figures:

  • American papers average 4.1 organizations per study, reflecting a highly distributed ecosystem spanning universities, hospitals, national laboratories, and private research institutions. Harvard Medical School leads in ML-enabled research volume, but the broader US landscape is remarkably decentralized.
  • Chinese papers average 2.6 organizations per study, indicating a more concentrated, high-throughput model where fewer institutions produce more output. China achieves its 43% global share through density rather than breadth, with 72.8 articles per university compared to 39.6 in the US.
  • Lastly, new collaboration corridors are emerging. India, Saudi Arabia, and the United States increasingly partner on applied sciences, materials research, engineering, and computer vision.

For AI practitioners building research relationships, these patterns matter. A collaboration strategy that works in the American context (distributed, multi-institutional) may need adjustment for Chinese partnerships (concentrated, institution-centric).

India’s quiet rise to third place

The report’s third-largest contributor doesn’t get enough attention: India has achieved a steep upward trajectory in ML-driven scientific output, now ranking behind only China and the United States.

What makes India’s approach distinctive is its focus on practical, scalable innovation rather than hype-driven experimentation. Indian ML research concentrates on socially relevant applications, like health diagnostics, climate resilience, fraud detection, agricultural optimization, and materials science, with local relevance.

Participation is broadening beyond elite institutions. Tier 1 and Tier 2 universities now contribute meaningfully to India’s ML research ecosystem, and a rapidly growing startup landscape is translating academic research into applied research and output.

For organizations looking to build global research partnerships, India represents an increasingly attractive option: strong technical talent, growing institutional capacity, and a pragmatic orientation toward problems that matter.

What this means for your ML strategy?

The gap discussed above mostly creates confusion, especially for practitioners who mistake visibility for impact.

If you are a researcher, don’t let trend-chasing distract from methodology that works. If Random Forest solves your problem, publish with confidence. The Nature portfolio clearly agrees.

If you are a practitioner, the tools that matter for production systems, like ensemble methods, classical feature engineering, and interpretable models, remain the tools that matter for scientific research. Investment in boring ML capabilities pays compounding returns.

If you are an ML organization, Geographic asymmetries in tool development versus research output suggest strategic opportunities. Countries consuming US-built frameworks may eventually develop alternatives.

If you are an investor or policymaker, the classical-vs-generative distribution should inform research priorities. Generative AI captures headlines, but traditional ML delivers the bulk of scientific impact.

ML is the infra layer of modern science

The takeaway is straightforward. Machine learning has stopped being the subject of research and has become the machinery behind it.

Scientists don’t write papers about Random Forest any more than they write papers about healthcare, for instance. They write papers about what Random Forest helped them discover.

As Asif Razzaq, Editor and Co-Founder of Marktechpost, put it: 

“This report shows that machine learning isn’t just reshaping AI, it’s reshaping science itself. The real story here is not hype, but impact, which tells that ML is now a fundamental instrument of modern scientific work.”

The 77% figure should humble anyone convinced that transformers and diffusion models define the field.

The geographic asymmetries should interest anyone thinking about how power flows through technical ecosystems. And the collaboration patterns should inform anyone building research networks in an increasingly multipolar scientific landscape.

The full interactive data is available at airesearchtrends.com, where you can explore country-level breakdowns, disciplinary distributions, and tool-by-tool analysis across 125+ countries and 5,000+ papers.

The next time someone tells you AI research is all about the latest foundation model, you’ll have 5,000 data points suggesting otherwise.

Thanks for reading!

This post was written by the Marktechpost team, specifically for Turing Post. We thank Marktechpost for sharing the ML Global Impact Report 2025 findings and supporting Turing Post’s mission to cut through the froth.

By , No comments:
Tags ,

ReBaz + Carpentries + RSE Conference

Mike's Notes

Plenty of opportunities in NZ to share research and upskill. I need to learn Python, and this might be a way to do so. I previously used Python with ESRI GIS, but that was 15 years ago.

Pipi 10 will have a Python interface.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

24/10/2025

ReBaz + Carpentries + RSE Conference

By: Mike Peters
On a Sandy Beach: 24/10/2025

Mike is the inventor and architect of Pipi and the founder of Ajabbi.

Research Bazaar

"The annual ResBaz (Research Bazaar) event – an amazing free series of online webinars and workshops that bring together researchers from around the country to develop skills in digital tools and research practices."

The next ResBaz Aotearoa 2026: 29 June - 3 July.

The Carpentries

" ..The Carpentries is a non-profit organisation that teaches foundational coding and data science skills to researchers worldwide. There are numerous courses freely available, including an introduction to Python for library and information workers."

HPC Carpentries

"HPC Carpentry teaches HPC-oriented coding, and data science skills to researchers. We want to work towards bringing High Performance Computing under the Carpentries umbrella."

New Zealand Research Software Engineering Conference

"Within the research sector, there is a growing number of people who combine expertise in programming with an intricate understanding of research. Although this combination of skills is extremely valuable, these people lack a formal place in the academic system. This means there is no easy way to recognise their contribution, to reward them, or to represent their views.  

A community-driven event 

In June 2020, NeSI decided to rebrand its successful Science Coding Conference to be named the NZ Research Software Engineers (RSE) Conference. Motivation for the change is two-fold:

  • to include people from all research communities who work on the cusp of technical and research domains, and
  •  to more fully align with the goals of the Australia / New Zealand RSE community. 

Initiated in the UK, the RSE movement is a global phenomenon with many associations now set up around the world. In line with that global trend, New Zealand’s community of research software users and developers has steadily grown, with more roles and numbers of people at the intersection of software and research." - NZRSE

By , No comments:
Tags , ,

No Science, No Startups: The Innovation Engine We're Switching Off

Mike's Notes

A great description of how science works in the US.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library > Subscriptions > Steve Blank
  • Home > Handbook > 

Last Updated

22/10/2025

No Science, No Startups: The Innovation Engine We're Switching Off

By: Steve Blank
On a Sandy Beach: 13/10/2025

Blank created the Lean Launchpad class and I-Corps curriculum which became the standard for science commercialization for the National Science Foundation, the National Institutes of Health and the U.S. Department of Energy. As of 2023, more than 3,051 teams and 1,300 startups have employed Blank’s methodologies..

Blank is co-creator of the U.S. Department of Defense's[6] Hacking for Defense program, and served on the Defense Business Board and the U.S. Navy’s Science and Technological Board. He is co-creator of the Gordian Knot Center for National Security Innovation at Stanford University.

Tons of words have been written about the Trump Administrations war on Science in Universities. But few people have asked what, exactly, is science? How does it work? Who are the scientists? What do they do? And more importantly, why should anyone (outside of universities) care?

(Unfortunately, you won’t see answers to these questions in the general press – it’s not clickbait enough. Nor will you read about it in the science journals– it’s not technical enough. You won’t hear a succinct description from any of the universities under fire, either – they’ve long lost the ability to connect the value of their work to the day-to-day life of the general public.)

In this post I’m going to describe how science works, how science and engineering have worked together to build innovative startups and companies in the U.S.—and why you should care.

(In a previous post I described how the U.S. built a science and technology ecosystem and why investment in science is directly correlated with a country’s national power. I suggest you read it first.)

How Science Works

I was older than I care to admit when I finally understood the difference between a scientist, an engineer, an entrepreneur and a venture capitalist; and the role that each played in the creation of advancements that made our economy thrive, our defense strong and America great.

Scientists

Scientists (sometimes called researchers) are the people who ask lots of questions about why and how things work. They don’t know the answers. Scientists are driven by curiosity, willing to make educated guesses (the fancy word is hypotheses) and run experiments to test their guesses. Most of the time their hypotheses are wrong. But every time they’re right they move the human race forward. We get new medicines, cures for diseases, new consumer goods, better and cheaper foods, etc.

Scientists tend to specialize in one area – biology, medical research, physics, agriculture, computer science, materials, math, etc. — although a few move between areas. The U.S. government has supported scientific research at scale (read billions of $s) since 1940.

Scientists tend to fall into two categories: Theorists and Experimentalists.

Theorists

Theorists develop mathematical models, abstract frameworks, and hypotheses for how the universe works. They don’t run experiments themselves—instead, they propose new ideas or principles, explain existing experimental results, predict phenomena that haven’t been observed yet. Theorists help define what reality might be.

Theorists can be found in different fields of science. For example:

 Science Study 
Physics  Quantum field theory, string theory, quantum mechanics
Biology Neuroscience and cognition, Systems Biology, gene regulation
Chemistry  Molecular dynamics, Quantum chemistry
Computer Science Design algorithms, prove limits of computation
Economics  Build models of markets or decision-making
Mathematics  Causal inference, Bayesian networks, Deep Learning

The best-known 20th-century theorist was Albert Einstein. His tools were a chalkboard and his brain. in 1905 he wrote an equation E=MC2 which told the world that a small amount of mass can be converted into a tremendous amount of energy. When he wrote it down, it was just theory. Other theorists in the 1930s and ’40s took Einstein’s theory and provided the impetus for building the atomic bomb. (Leo Szilard conceived neutron chain reaction idea, Hans Bethe led the Theoretical Division at Los Alamos, Edward Teller developed hydrogen bomb theory.) Einstein’s theory was demonstrably proved correct over Hiroshima and Nagasaki.

Experimentalists

In addition to theorists, other scientists – called experimentalists – design and run experiments in a lab. The pictures you see of scientists in lab coats in front of microscopes, test tubes, particle accelerators or NASA spacecraft are likely experimentalists. They test hypotheses by developing and performing experiments. An example of this would be NASA’s James Webb telescope or the LIGO Gravitational-Wave Observatory experiment. (As we’ll see later, often it’s engineers who build the devices the experimentalists use.)

Some of these experimentalists focus on Basic Science, working to get knowledge for its own sake and understand fundamental principles of nature with no immediate practical use in mind.

Other experimentalists work in Applied Science, which uses the findings and theories derived from Basic Science to design, innovate, and improve products and processes.

Applied scientists solve practical problems oriented toward real-world applications. (Scientists at Los Alamos weretrying to understand the critical mass of U-235 (the minimum amount that would explode.) Basic science lays the groundwork for breakthroughs in applied science. For instance: Quantum mechanics (basic science) led to semiconductors which led to computers (applied science). Germ theory (basic science) led to antibiotics and vaccines (applied science). In the 20th century Applied scientists did not start the companies that make end products. Engineers and entrepreneurs did this. (In the 21st century more Applied Scientists, particularly in life sciences, have also spun out companies from their labs.)

Scientists

Where is Science in the U.S. Done?

America’s unique insight that has allowed it to dominate Science and invention, is that after WWII we gave Research and Development money to universities, rather than only funding government laboratories. No other country did this at scale.

Corporate Research Centers

In the 20th century, U.S. companies put their excess profits into corporate research labs. Basic research in the U.S. was done in at Dupont, Bell Labs, IBM, AT&T, Xerox, Kodak, GE, et al.

This changed in 1982, when the Securities and Exchange Commission ruled that it was legal for companies to buy their own stock (reducing the number of shares available to the public and inflating their stock price.) Very quickly Basic Science in corporate research all but disappeared. Companies focused on Applied Research to maximize shareholder value. In its place, Theory and Basic research is now done in research universities.

Research Universities

From the outside (or if you’re an undergraduate) universities look like a place where students take classes and get a degree. However, in a research university there is something equally important going on. Science faculty in these schools not only teach, but they are expected to produce new knowledge—through experiments, publications, patents, or creative work. Professors get grants and contracts from federal agencies (e.g., NSF, NIH, DoD), foundations, and industry. And the university builds Labs, centers, libraries, and advanced computing facilities that support these activities.

In the U.S. there are 542 research universities, ranked by the Carnegie Classification into three categories.

R1: 187 Universities – Very High Research Activity

  • Conduct extensive research and award many doctoral degrees.
  • Examples: Stanford, UC Berkeley, Harvard, MIT, Michigan, Texas A&M …

R2: 139 Universities – High Research Activity

  • Substantial but smaller research scale.
  • Examples: Baylor, Wake Forest, UC Santa Cruz, …

R3: 216 Research Colleges/Universities

  • Limited research focus; more teaching-oriented doctoral programs.
  • Smaller state universities

Why Universities Matter to Science

U.S. universities perform about 50% of all basic science research (physics, chemistry, biology, social sciences, etc.) because they are training grounds for graduate students and postdocs. Universities spend ~$109 billion a year on research. ~$60 billion of that $109 billion comes from the National Institutes for Health (NIH) for biomedical research, National Science Foundation (NSF) for basic science, Department of War (DoW), Department of Energy (DOE), for energy/physics/nuclear, DARPA, NASA. (Companies tend to invest in applied research and development, that leads directly to saleable products.)

Professors (especially in Science, Technology, Engineering and Math) run labs that function like mini startups. They ask research questions, then hire grad students, postdocs, and staff and write grant proposals to fund their work, often spending 30–50% of their time writing and managing grants. When they get a grant the lead researcher (typically a faculty member/head of the lab) is called the Principal Investigator (PI).

The Labs are both workplaces and classrooms. Graduate students and Postdocs do the day-to-day science work as part of their training (often for a Ph.D.). Postdocs are full-time researchers gaining further specialization. Undergraduates may also assist in research, especially at top-tier schools.

(Up until 2025, U.S. science was deeply international with ~40–50% of U.S. basic research done by foreign-born researchers (graduate students, postdocs, and faculty). Immigration and student visas were a critical part of American research capacity.)

The results of this research are shared with the agencies that funded it, published in journals, presented at conferences and often patented or spun off into startups via technology transfer offices. A lot of commercial tech—from Google search to CRISPR—started in university labs.

Universities support their science researchers with basic administrative staff (for compliance, purchasing, and safety) but uniquely in the U.S., by providing the best research facilities (labs, cleanrooms, telescopes), and core scientific services: DNA sequencing centers, electron microscopes, access to cloud, data analysis hubs, etc. These were the best in the world – until the sweeping cuts in 2025.

Engineers Build on the Work of Scientists

Engineers design and build things on top of the discoveries of scientists. For example, seven years after scientists split the atom, it took 10s of thousands of engineers to build an atomic bomb. From the outset, the engineers knew what they wanted to build because of the basic and applied scientific research that came before them.

Scientists Versus Engineers

Engineers create plans, use software to test their designs, then… cut sheet metal, build rocket engines, construct buildings and bridges, design chips, build equipment for experimentalists, design cars, etc.

As an example, at Nvidia their GPU chips are built in a chip factory (TSMC) using the Applied science done by companies like Applied Materials which in turn is based on Basic science of semiconductor researchers. And the massive data centers OpenAI, Microsoft, Google, et al that use Nvidia chips are being built by mechanical and other types of engineers.

My favorite example is that the reusable SpaceX rocket landings are made possible by the Applied Science research on Convex Optimization frameworks and algorithms by Steven Boyd of Stanford. And Boyd’s work was based on the Basic science mathematical field of convex analysis (SpaceX, NASA, JPL, Blue Origin, Rocket Lab all use variations of Convex Optimization for guidance, control, and landing.)

Startup Entrepreneurs Build Iteratively and Incrementally

Entrepreneurs build companies to bring new products to market. They hire engineers to build, test and refine products.

Engineers and entrepreneurs operate with very different mindsets, goals, and tolerances for risk and failure. (Many great entrepreneurs start as engineers e.g., Musk, Gates, Page/Brin). An engineer’s goal is to design and deliver a solution to a known problem with a given set of specifications.

In contrast, entrepreneurs start with a series of unknowns about who are the customers, what are the wanted product features, pricing, etc. They retire each of these risks by building an iterative series of minimum viable products to find product/market fit and customer adoption. They pivot their solution as needed when they discover their initial assumptions are incorrect. (Treating each business unknown as a hypothesis is the entrepreneurs’ version of the Scientific Method.)

Venture Capitalists Fund Entrepreneurs

Venture capitalists (VCs) are the people who fund entrepreneurs who work with engineers who build things that applied scientists have proven from basic researchers.

Unlike banks which will give out loans for projects that have known specifications and outcomes, VCs invest in a portfolio of much riskier investments. While banks make money on the interest they charge on each loan, VCs take part ownership (equity) in the companies they invest in. While most VC investments fail, the ones that succeed make up for that.

Most VCs are not scientists. Few are engineers, some have been entrepreneurs. The best VCs understand technical trends and their investments help shape the future. VCs do not invest in science/researchers. VCs want to minimize the risk of their investment, so they mostly want to take engineering and manufacturing risk, but less so on applied science risk and rarely on basic research risk. Hence the role of government and Universities.

VCs invest in projects that can take advantage of science and deliver products within the time horizon of their funds (3–7 years). Science often needs decades before a killer app is visible.

As the flow of science-based technologies dries up, the opportunities for U.S. venture capital based on deep tech will decline, with its future in countries that are investing in science – China or Europe.

Why Have Scientists? Why Not Just a Country of Engineers, Entrepreneurs and VCs (or AI)?

If you’ve read so far, you might be scratching your head and asking, “Why do we have scientists at all? Why pay for people to sit around and think? Why spend money on people who run experiments when most of those experiments fail? Can’t we replace them with AI?”

The output of this university-industry-government science partnership became the foundation of Silicon Valley, the aerospace sector, the biotechnology industry, Quantum and AI. These investments gave us rockets, cures for cancer, medical devices, the Internet, Chat GPT, AI and more.

Investment in science is directly correlated with national power. Weaken science, you weaken the long-term growth of the economy, and national defense.

Tech firms’ investments of $100s of billions in AI data centers is greater than the federal government’s R&D expenditures. But these investments are in engineering not in science. The goal of making scientists redundant using artificial general intelligence misses the point that AI will (and is) making scientists more productive – not replacing them.

Countries that neglect science become dependent on those that don’t. U.S. post-WWII dominance came from basic science investments (OSRD, NSF, NIH, DOE labs). After WWII ended, the UK slashed science investment which allowed the U.S. to commercialize the British inventions made during the war.

The Soviet Union’s collapse partly reflected failure to convert science into sustained innovation, during the same time that U.S. universities, startups and venture capital created Silicon Valley. Long-term military and economic advantage (nuclear weapons, GPS, AI) trace back to scientific research ecosystems.

Lessons Learned

  • Scientists come in two categories
    • Theorists and experimentalists
    • Two types of experimentalists; Basic science (learn new things) or applied science (practical applications of the science)
    • Scientists train talent, create patentable inventions and solutions for national defense
  • Engineers design and build things on top of the discoveries of scientists
  • Entrepreneurs test and push the boundaries of what products could be built
  • Venture Capital provides the money to startups
  • Scientists, engineers, entrepreneurs – these roles are complementary
    • Remove one and the system degrades
  • Science won’t stop
    • Cut U.S. funding, then science will happen in other countries that understand its relationship to making a nation great – like China.
    • National power is derived from investments in Science
    • Reducing investment in basic and applied science makes America weak

 Appendix – How Does Science Work? – The Scientific Method

Whether you were a theorist or experimentalist, for the last 500 years the way to test science was by using the scientific method. This method starts by a scientist wondering and asking, “Here’s how I think this should work, let’s test the idea.”

The goal of the scientific method is to turn a guess (in science called a hypothesis) into actual evidence. Scientists do this by first designing an experiment to test their guess/hypothesis. They then run the experiment and collect and analyze the result and ask, “Did the result validate, invalidate the hypothesis? Or did it give us completely new ideas?” Scientists build instruments and run experiments not because of what they know, but because of what they don’t know.

These experiments can be simple ones costing thousands of dollars that can be run in a university biology lab while others may require billions of dollars to build a satellite, particle accelerator or telescope. (The U.S. took the lead in Science after WWII when the government realised that funding scientists was good for the American economy and defence.)

Good science is reproducible. Scientists just don’t publish their results, but they also publish the details of how they ran their experiment. That allows other scientists to run the same experiment and see if they get the same result for themselves. That makes the scientific method self-correcting (you or others can see mistakes).

One other benefit of the scientific method is that scientists (and the people who fund them) expect most of the experiments to fail, but the failures are part of learning and discovery. They teach us what works and what doesn’t. Failure in science testing unknowns means learning and discovery.

By , No comments:
Tags ,

Data Colada Table of Contents

Mike's Notes

Data Colada is a remarkable effort by Uri Simonsohn, Leif Nelson and Joe Simmons on topics such as fake data, research design, meta-analysis, and the reproducibility of science, among others.

Here is the table of contents of Data Colada.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

26/09/2025

Data Colada Table of Contents

By: Uri Simonsohn, Leif Nelson and Joe Simmons.
Data Colada: 26/09/2025

Thinking about evidence and vice versa.

Table of Contents

About Research Design

About Research Tips

Comment on media coverage

Credibility Lab

Data Replicada

Discuss own paper

Discuss Paper by Others

Effect size

Fake data

file-drawer

Interactions

Just fun

Lawsuits

Meta Analysis

Music

On Bayesian Stats

Opinion

p-curve

p-hacking

Preregistration

Replication

Reproducibility

Researchbox

Statistical Power

Teaching

Unexpectedly Difficult Statistical Concepts

By , No comments:
Tags , , ,
Subscribe to: Posts (Atom)