Showing posts with label mathematics. Show all posts
Showing posts with label mathematics. Show all posts

How to Build and Optimize High-Performance Deep Neural Networks from Scratch

Mike's Notes

Another fascinating article by Vincent Granville.

Resources

References

  • Intuitive Machine Learning by Vincent Granville.

Repository

  • Home > Ajabbi Research > Library > Subscriptions > MLtechniques AI Newsletter
  • Home > Handbook > 

Last Updated

14/09/2026

How to Build and Optimize High-Performance Deep Neural Networks from Scratch

By: Vincent Granville
MLtechniques AI Newsletter: 20/12/2025

Vincent Granville: Vincent Granville is a well-known pioneering AI scientist and machine learning expert, Chief AI Architect at BondingAI, author and patents owner with some related to trustworthiness scores. Vincent worked with Visa (credit card fraud), Wells Fargo, eBay (Google keyword campaigns), NBC, Microsoft, and CNET. He is currently working on no-Blackbox, auditable, hallucination-free secure Enterprise AI requiring no GPU and offering relevancy and trustworthiness scores in prompt results. Also, Vincent recently developed new, explainable deep neural network models along with distillation-resistant watermarking technology for model and data protection, to detect unauthorized uses.

Vincent is also a former post-doc at Cambridge University, and the National Institute of Statistical Sciences (NISS). He published in IEEE Transactions on Pattern Analysis and Machine Intelligence (500+ citations), Journal of Number Theory, and Journal of the Royal Statistical Society (Series B). He is the author of multiple books, available here, including “Synthetic Data and Generative AI” (Elsevier, 2024). Vincent lives in Washington state, and enjoys doing research on stochastic processes, dynamical systems, experimental math and probabilistic number theory.

...

With explainable AI, intuitive parameters easy to fine-tune, versatile, robust, fast to train, without any library other than Numpy. In short, you have full control over all components, allowing for deep customization, and much fewer parameters than in standard architectures.

Introduction

I explore deep neural networks (DNNs) starting from the foundations, introducing a new type of architecture, as much different from machine learning than it is from traditional AI. The original adaptive loss function introduced here for the first time, leads to spectacular performance improvements via a mechanism called equalization.

To accurately approximate any response, rather than connecting neurons with linear combinations and activation between layers, I use non-linear functions without activation, reducing the number of parameters, leading to explainability, easier fine tune, and faster training. The adaptive equalizer – a dynamical subsystem of its own – eliminates the linear part of the model, focusing on higher order interactions to accelerate convergence.

One example involves the Riemann zeta function. I exploit its well-known universality property to approximate any response. My system also handles singularities to deal with rare events or fraud detection. The loss function can be nowhere differentiable such as a Brownian motion. Many of the new discoveries are applicable to standard DNNs. Built from scratch, the Python code does not rely on any library other than Numpy. In particular, I do not use PyTorch, TensorFlow or Keras.

10 Key Features to Boost DNN Performance

Here is a high-level summary of key features that boost performance, in simple English. Most apply to any deep neural network. Also, the focus is on the core engine that powers all DNNs: gradient descent, layering and loss function.

  • Reparameterization — Typically, in DNNs, many different parameter sets lead to the same optimum: loss minimization. DNN models are non-identifiable. This redundancy is a strength that increases the odds of landing on a good solution. You can change the parameterization to reduce redundancy and increase identifiability. You achieve this by reparameterization and eliminating intermediate layers. It usually does not improve the results. Or you can do the opposite: add redundant layers to increase leeway. Or you can transform parameters while keeping them in the same range. For instance, use θ’ = θ2 instead of θ, both in [0, 1]. This flexibility allows you to achieve better results but require testing.
  • Ghost parameters — Adding artificial, non-necessary or redundant parameters can make the descent more fluid. It gives more chances (more potential paths) for the gradient descent to end in a good configuration. You may use some of these ghost parameters for watermarking your DNN, to protect your model against hijacking and unauthorized use.
  • Layer flattening — Instead of hierarchical layers, you can optimize the entire structure at once, across all layers. That is, minimizing the loss function globally at each epoch rather than propagating changes back and forth throughout many layers. It reduces error propagation and eliminates the need for explicit activation functions. It may reduce the risk of getting stuck (gradient vanishing).
  • Sub-layers — In some sense, it is the oppositive of layer flattening. At each epoch, you minimize the loss iteratively, for a subset of parameters (sub-layer) at a time, keeping the other parameters unchanged. It works as in the EM algorithm. Each parameter subset optimization is a sub-epoch within an epoch. A full epoch consists of going through the full set of parameters. This technique is useful in high dimensional problems with complex, high-redundancy parameter structure.
  • Swarm optimization — Handy for lower dimensional problems with singularities and non-differentiable loss function. You start with multiple random initial configurations (called particles) in the gradient descent. For each particle, you explore random neighbors in the parameter space and move towards the neighbor that minimizes the loss. You need a good normalization of the learning rate to make it work. Consider working with adaptive learning rates that depend on the epoch and/or axis (different learning rate for each axis, that is, for each parameter).
  • Decaying entropy — Instead of descending all the time, allow for occasional random ascents in the gradient descent. Globally or for specific parameters chosen randomly. Especially in the earlier epochs. Think of it like a cave exploration: to reach the very bottom of a deep cave, sometimes you need to go up through a siphon and climb back up from it on the other side to be able to further go down. Otherwise, you get stuck in the siphon. The hyperparameter controlling the ups and downs is the temperature. At zero, the descent is a pure descent with no ups. I call it chaotic gradient descent as it is different from stochastic descent. See Figure 2.
  • Adaptive loss — Some knobs turn the loss function into one that changes over time. In one case, I force the loss function to converge towards the model evaluation metric via a number of transitions over time. These changes in the loss function typically take place when the gradient descent stops improving, re-accelerating the descent to move away from a local minimum. See Figure 1.
  • Equalization — In my implementation, this is the feature that led to the biggest improvement: reducing the number of epochs thus accelerating convergence along with eliminating a number of gradient descent issues. It consists of replacing the fixed response y at each epoch by an adaptive response y’ = φ(y, α). Here φ belongs to a parametric family of transforms indexed by α, and stable under composition and inversion. For instance, scale-location transforms. So, even if you don’t know the history of the successive transforms with a different data-driven α at each epoch, it is very easy at the end to apply an inverse transform to map the final response back to its original, via ordinary least squares. You apply the same inverse transform to predicted values.
  • Normalized parameters — In my implementation all parameters are in [0, 1]. Also true for the input data after normalization. To use parameters outside that range, use reparameterization, for instance θ’ = 1 / (1 – θ) while optimizing for θ in [0, 1]. This approach eliminates gradient explosion.
  • Math-free gradient — No need to use chain rules and other math formula to compute the partial derivatives in the gradient. Indeed, you don’t even need to know the mathematical expression of the loss function. It works on pure data without math functions. Numerical precision is critical. Some math functions are pre-computed in 0.0001 increments and stored as a table. This is possible because the argument (a parameter) is in [0, 1]. It can significantly increase speed.

Parameters in my model have an intuitive interpretation, such as centers and skewness, corresponding to kernels in high-dimensional adaptive kernel density estimation or mixtures of Gaussians that can model any type of response. In some cases, the unknown (predicted) centers are expected and visible in the response such as in clustering problems. Sometimes — depending on the settings — they are not and act as hidden or latent parameters.

Figure 1: Loss function updated whenever the gradient descent stalls

Figure 2: Chaotic descent controlled by temperature knob

Finally, while I implicitly use tensors, you don’t need to know what a tensor is and there is no call to libraries such as TensorFlow.

Get the Full Code and Documentation

The PDF with many illustrations is available as paper 55, here. It also features the Python code (with link to GitHub), the replicable data generated by the code, the theory, and various options including for evaluation. The blue links in the PDF are clickable once you download the document from GitHub and view it in any browser. Keywords highlighted in orange are index keywords.

Figure 3: Descent with swarm optimization

Conclusions

This original non-standard DNN architecture turns black boxes into explainable AI with intuitive parameters. The focus is on speed (fewer parameters, faster convergence), robustness, and versatility with easy, rapid fine-tune and many options. It does not use PyTorch or similar libraries: you have full control over the code, increasing security and customization. Tested on synthetic data batches for predictive analytics, high dimensional curve fitting and noise filtering in various settings, it incorporates many innovative features. Some of them, like the equalizer, have a dramatic impact on performance and can also be implemented in standard DNNs. The core of the code is simple and consists of fewer than 200 lines, relying on Numpy alone.

Figure 4: One of many use cases: predicting orbits

By , No comments:
Tags , , ,

Making a start on Hilbert Spaces and matrix calculations using CPU

Mike's Notes

Working notes on finding a way for Pipi to use genetic algorithms with Hilbert Spaces and matrix calculations. A desktop exercise with Google at the moment. Computational experiments on CPU to come. Some generated Python and CFML sample code below to start playing with. 

It looks promising; it might help address limitations of other methods for using fitness landscapes. The output curves do look better.

This might void the need to use a Cerebras wafer for these fitness landscape calculations.

The next step is to watch free maths lectures on YouTube, then do a lot of drawings and process diagrams. Then play with code and see what happens. 😎

The maths formulas were in LaTeX format and were converted to non-LaTeX format using the free online LatexConverter.

Note

Nothing has been verified or tested. 😎

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library >
  • Home > Handbook > 

Last Updated

03/09/2026

Making a start on Hilbert Spaces and matrix calculations using CPU

By: Mike Peters & Google Search AI Mode
On a Sandy Beach: 30/08/2026

Mike Peters: Mike is the inventor and architect of Pipi and the founder of Ajabbi.

Google Search AI Mode: Free and surprisingly good.

...

Hilbert Spaces

Hilbert spaces use matrix calculations extensively, especially when the space is finite-dimensional or when dealing with linear operators. Here is how they connect:

  1. Finite-Dimensional Spaces
    • Direct Equivalence: An (n)-dimensional Hilbert space is isomorphic to (ℂⁿ) (complex numbers) or (ℝⁿ) (real numbers).
    • Vectors: Elements in the space are represented as column vectors.
    • Operators: Linear operators acting on the space are represented as (n × n) matrices.
    • Inner Product: The inner product is calculated using matrix multiplication (specifically, the conjugate transpose of a vector multiplied by another vector).
  2. Infinite-Dimensional Spaces
    • Infinite Matrices: In infinite-dimensional Hilbert spaces (like L²), the space of square-integrable functions, operators can be represented as infinite matrices.
    • Basis Dependence: By choosing an orthonormal basis, any linear operator can be written as a matrix with infinitely many rows and columns.
  3. Quantum Mechanics (Matrix Mechanics)
    • State Vectors: Quantum states are vectors in a Hilbert space.
    • Observables: Physical properties (like position, momentum, or spin) are Hermitian operators.
    • Heisenberg's Formulation: Werner Heisenberg's formulation of quantum mechanics is literally called "matrix mechanics" because it uses matrix calculations to solve Hilbert space problems.

In machine learning, Hilbert spaces—specifically Reproducible Kernel Hilbert Spaces (RKHS)—rely heavily on matrix calculations to make complex, non-linear data linearly separable.

Instead of explicitly mapping data points to an infinite-dimensional space, ML algorithms use the Kernel Trick to compute everything via an N × N matrix, where N is the number of data samples.

The Gram (Kernel) Matrix

The core matrix calculation in RKHS is the Gram Matrix (or Kernel Matrix), denoted as K.

  • Formula: (Kᵢⱼ = k(xᵢ, xⱼ) = ⟨ Φ(xᵢ), Φ(xⱼ) ⟩_{H})
  • Meaning: Each entry i,j is the inner product of data points (xᵢ) and (xⱼ) in the Hilbert space (H).
  • Properties: It is always a symmetric and positive semi-definite (PSD) matrix.

Core Matrix Operations in RKHS

Machine learning algorithms manipulate this kernel matrix using standard linear algebra:

  • Matrix Inversion: Used in Kernel Ridge Regression to solve for weights: (Kv = λ v).
  • Eigenvalue Decomposition: Used in Kernel PCA to find non-linear principal components by solving (Kv = λ v).
  • Quadratic Programming: Used in Support Vector Machines (SVMs) to maximise margins using matrix-vector products involving (K).

Computational Bottlenecks

Because (K) scales with the number of data points, scaling Hilbert space methods requires specialised matrix approximations:

  • Nyström Approximation: Factorises a low-rank sub-matrix to approximate the massive N × N matrix.
  • Random Fourier Features: Avoids the exact matrix calculation entirely by mapping data into a randomised, lower-dimensional space.

Graph image

Genetic Algorithms

Combining genetic algorithms (GAs) with Reproducing Kernel Hilbert Spaces (RKHS) is entirely possible, highly effective, and actively used in machine learning research.

In this hybrid framework, the fitness landscape is modelled as a continuous function belonging to an RKHS. This approach addresses a major limitation of traditional GAs: evaluating an individual's true fitness can be computationally expensive (e.g., running a physics simulation or a wet-lab experiment). The RKHS serves as an intelligent surrogate model.

Graph image

Core Mathematical Engines At Work

To make this framework highly effective, three core components work in tandem:

  • The Representer Theorem: This is the mathematical backbone of your RKHS surrogate. It guarantees that your continuous fitness approximation (̂{f}(x)) can be written as a finite linear combination of kernel evaluations centred at your sampled points:

(ᶠ(x)=∑ ᵢ₌₁ᴺα ᵢK(x,xᵢ))

  • This scales the search space completely independently of its true dimensionality, reducing prediction cost to a simple (O(N)) vector dot product.
  • Informed Exploration vs. Exploitation: Instead of letting the GA evaluate individuals blindly on the surrogate, you can use the RKHS variance (uncertainty) to construct an Acquisition Function (like Expected Improvement or Upper Confidence Bound). The GA then maximises this acquisition function rather than the raw estimated fitness, forcing the algorithm to intelligently search unmapped areas of the landscape.
  • Gram Matrix Regularisation: As new true data points are evaluated, they enter the Gram Matrix. To prevent numerical instability or overfitting as (N) grows, a small regularisation ridge ((λ I)) is maintained, smoothly adjusting the landscape's rigidity without rebuilding the framework from scratch.

How the Combination Works

[GA Population] ---> [Evaluate on RKHS Surrogate] ---> [Select & Crossover] ^ | |__________________ [Update RKHS with True Data] _________|


  1. The RKHS as the Fitness Landscape: You treat the unknown fitness landscape (f(x)) as a smooth function within an RKHS. By evaluating a small set of initial points, you use Kernel Ridge Regression or Kriging (Gaussian Process Regression) to build a continuous, analytical approximation of the landscape.
  2. Genetic Search on the Surrogate: The GA searches this approximated RKHS landscape. Because evaluating the RKHS kernel matrix is mathematically cheap (O(N)) for a new prediction, the GA can evolve through thousands of generations in seconds.
  3. Adaptive Sampling (Bayesian Optimisation): The best candidates found by the GA are evaluated using the true, expensive fitness function. These new data points are added back into the Gram matrix, updating the RKHS landscape to make it more accurate where it matters most.

Key Advantages of This Hybrid Approach

  • Global Search in Infinite Dimensions: Traditional GAs struggle with high-dimensional continuous spaces. By mapping the search space into an RKHS via a kernel (like the Gaussian RBF kernel), the GA implicitly searches a highly expressive, infinite-dimensional feature space.
  • Guaranteed Smoothness: Functions in an RKHS are mathematically smooth and bounded [6]. This eliminates extreme, discontinuous noise in the fitness landscape, allowing the GA to find gradient-like paths toward global optima without getting trapped in microscopic local minima.
  • Exact Kernel Alignment: You can use a GA to optimise the hyperparameters of the RKHS kernel itself (e.g., finding the optimal length-scale (σ ) of an RBF kernel) to perfectly align the geometry of the Hilbert space with your actual problem landscape.

Real-World Use Cases

  • Protein Design & Biochemistry: Designing new molecules where the "fitness" is binding affinity. The RKHS models the chemical landscape based on known proteins, and the GA mutates amino acid sequences to find optimal binders.
  • Hyperparameter Optimisation: Optimising complex neural network architectures where the evaluation landscape is highly non-linear and expensive to compute.
  • Robotics Control: Evolving locomotion parameters for robotics where physical testing is slow or dangerous.

Implementation Challenges

  • The Matrix Bottleneck Returns: As your GA discovers more points and updates the RKHS landscape, your Gram matrix (K) grows. Inverting this matrix to update the landscape scales at O(N³). You will need to use Nyström approximations or sparse online kernel methods if your total evaluation points exceed 10,000.
  • Exploration vs. Exploitation: GAs are aggressive exploiters. If your RKHS surrogate model is inaccurate in an unexplored region, the GA might get stuck optimising a "fake" peak. You must include an uncertainty metric (like the variance in a Gaussian Process) to force the GA to explore unknown areas.

Kernel Ridge Regression or Kriging (Gaussian Process Regression)

Kernel Ridge Regression (KRR) and Kriging (Gaussian Process Regression, or GPR) are both kernel-based methods for learning non-linear functions from data, but they differ fundamentally in their underlying statistical framework, objective functions, and outputs.

Key Differences

  • Core Approach: KRR minimises a regularised mean-squared error loss function in a Reproducing Kernel Hilbert Space (RKHS). GPR uses a probabilistic (Bayesian) approach, defining a Gaussian process prior over functions and updating it with a likelihood function based on observed data.
  • Uncertainty Estimation: KRR outputs point predictions only. GPR (and Kriging) naturally quantifies uncertainty, providing full posterior distributions, variance estimates, and confidence intervals.
  • Hyperparameter Optimisation: KRR typically optimises kernel parameters using grid search with cross-validation on a loss function. GPR optimises hyperparameters via gradient ascent on the marginal likelihood.
  • Terminology & Origin: Kriging originated in geostatistics (mining) to find the Best Linear Unbiased Predictor (BLUP), whereas GPR stems from machine learning and stochastic processes. Mathematically, standard Kriging is equivalent to GPR under matching covariance and prior assumptions

When to Use Which

  • Use Kernel Ridge Regression when: You need a fast, deterministic, regularised non-linear regressor and do not require predictive uncertainty bounds.
  • Use Kriging / Gaussian Process Regression when: You need confidence intervals for predictions, want to optimise hyperparameters via marginal likelihood, or are modelling spatial/geostatistical data.

Python Code (not tested)

Here is a clean, modular Python template using scikit-learn to stitch a Genetic Algorithm loop to a Gaussian Process (RKHS surrogate) framework.

import numpy as np
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import Matern

# --- 1. CONFIGURATION & SIMULATION BACKEND ---
BOUNDS = np.array([[-5.0, 5.0], [-5.0, 5.0]])  # 2D search space boundaries
N_DIM = BOUNDS.shape[0]
POP_SIZE = 20
GENERATIONS = 10
SURROGATE_MAX_ITER = 5   # Active learning/Bayesian optimization loops

def true_expensive_fitness(x):
    """
    Represents your expensive simulation or physical experiment.
    Expects a 1D array of shape (N_DIM,). Returns a scalar.
    """
    # Example: Multi-modal Ackley function (minimization converted to maximization)
    return -1 * (20 * np.exp(-0.2 * np.sqrt(0.5 * np.sum(x**2))) + 
                 np.exp(0.5 * np.sum(np.cos(2 * np.pi * x))) - 20 - np.e)

# --- 2. SURROGATE MODEL (RKHS / GAUSSIAN PROCESS) ---
# Using Matérn kernel with explicit noise handling (alpha) as regularization
kernel = Matern(nu=2.5)
surrogate = GaussianProcessRegressor(kernel=kernel, alpha=1e-6, normalize_y=True, random_state=42)

# --- 3. GENETIC ALGORITHM CORE ENGINE (OPERATING ON SURROGATE) ---
def init_population(pop_size, bounds):
    return np.random.uniform(bounds[:, 0], bounds[:, 1], size=(pop_size, len(bounds)))

def crossover(parent1, parent2):
    # Blend crossover (BLX-alpha)
    alpha = 0.5
    gamma = (1 + 2 * alpha) * np.random.random(size=N_DIM) - alpha
    return parent1 + gamma * (parent2 - parent1)

def mutate(individual, bounds, rate=0.2, scale=0.5):
    for i in range(len(individual)):
        if np.random.random() < rate:
            individual[i] += np.random.normal(0, scale)
            individual[i] = np.clip(individual[i], bounds[i, 0], bounds[i, 1])
    return individual

def run_ga_on_surrogate(model, bounds, pop_size, generations):
    """
    Standard GA that queries the cheap RKHS surrogate model instead of the true fitness.
    """
    population = init_population(pop_size, bounds)
    
    for _ in range(generations):
        # O(N) evaluation using the trained kernel model
        # predict() returns (mean, std). We maximize the predicted mean fitness.
        fitnesses, _ = model.predict(population, return_std=True)
        
        # Rank-based selection
        idx = np.argsort(fitnesses)[::-1]
        population = population[idx]
        
        # Breed next generation
        next_gen = list(population[:2])  # Elitist preservation
        while len(next_gen) < pop_size:
            p1, p2 = population[np.random.randint(0, 5)], population[np.random.randint(0, 5)]
            child = crossover(p1, p2)
            child = mutate(child, bounds)
            next_gen.append(child)
            
        population = np.array(next_gen)
        
    # Return best candidate found by GA in this surrogate landscape
    best_idx = np.argmax(model.predict(population, return_std=False))
    return population[best_idx]

# --- 4. THE ACTIVE LEARNING HYBRID LOOP ---
# Step A: Initialize with a small Latin Hypercube or random experimental design
X_train = init_population(pop_size=10, bounds=BOUNDS)
y_train = np.array([true_expensive_fitness(x) for x in X_train])

print("Starting Hybrid GA-RKHS Optimization Engine...\n")

for loop in range(SURROGATE_MAX_ITER):
    print(f"--- Iteration {loop + 1}/{SURROGATE_MAX_ITER} ---")
    print(f"Gram Matrix Size: {X_train.shape[0]} points")
    
    # Step B: Fit/Update RKHS surrogate model
    surrogate.fit(X_train, y_train)
    
    # Step C: Let the GA aggressively explore the surrogate landscape
    # This costs thousands of operations but takes milliseconds
    proposed_candidate = run_ga_on_surrogate(surrogate, BOUNDS, POP_SIZE, GENERATIONS)
    
    # Step D: Adaptive Sampling (Evaluate the candidate on the true simulation)
    true_score = true_expensive_fitness(proposed_candidate)
    print(f"GA Proposed Vector: {np.round(proposed_candidate, 4)}")
    print(f"True Fitness Score: {true_score:.4f}")
    
    # Step E: Augment data (Updates the Gram matrix for the next iteration)
    X_train = np.vstack([X_train, proposed_candidate])
    y_train = np.append(y_train, true_score)
    print(f"Current Best Score Found: {np.max(y_train):.4f}\n")

print("Optimization Complete.")

Enhancements 

  • Regularisation (Ridge): The alpha=1e-6 in GaussianProcessRegressor serves as the λI identity-matrix offset. This prevents numerical inversion crashes if the GA proposes points structurally close to existing data.
  • Normalisation Engine: normalize_y=True transforms unstable fitness functions to a standard normal distribution mapping, stabilising kernel hyperparameter tuning during .fit().
  • Scalability Hand-off: run_ga_on_surrogate can run for massive populations or generations; it never interacts with true_expensive_fitness, protecting your heavy computation budget.

CFML Code (not tested)

<cfscript>
/**
 * CFML Hybrid GA-RKHS Optimization Engine
 */

// --- 1. CONFIGURATION & CONFIG MATRIX ---
variables.bounds = [ {min: -5.0, max: 5.0}, {min: -5.0, max: 5.0} ]; // 2D Search Space
variables.popSize = 20;
variables.generations = 10;
variables.surrogateMaxIter = 5;

/**
 * Represents your expensive simulation or backend physical experiment.
 * Minimisation problem converted to Maximisation.
 */
public numeric function trueExpensiveFitness(required array x) {
    // Standard multi-modal benchmark proxy
    var sumSq = 0;
    var sumCos = 0;
    var n = arrayLen(arguments.x);
    
    for (var i = 1; i <= n; i++) {
        sumSq += arguments.x[i] ^ 2;
        sumCos += cos(2 * pi() * arguments.x[i]);
    }
    
    var term1 = -20.0 * exp(-0.2 * sqrt(0.5 * sumSq));
    var term2 = -exp(0.5 * sumCos);
    return -1 * (term1 + term2 - 20 - exp(1));
}

// --- 2. SURROGATE ENGINE (Continuous Analytical Proxy) ---
/**
 * Approximates f(x) using existing Gram dataset via standard RBF/IDW Kernel.
 */
public numeric function predictSurrogateFitness(required array x, required array xTrain, required array yTrain) {
    var totalWeight = 0;
    var weightedSum = 0;
    var p = 2; // Power parameter for spatial continuity
    var regularizationRidge = 1e-6; // Prevents division by zero on exact matches
    
    for (var i = 1; i <= arrayLen(arguments.xTrain); i++) {
        var distSq = 0;
        for (var d = 1; d <= arrayLen(arguments.x); d++) {
            distSq += (arguments.x[d] - arguments.xTrain[i][d]) ^ 2;
        }
        var dist = sqrt(distSq) + regularizationRidge;
        var weight = 1.0 / (dist ^ p);
        
        totalWeight += weight;
        weightedSum += weight * arguments.yTrain[i];
    }
    
    return weightedSum / totalWeight;
}

// --- 3. GENETIC ALGORITHM CORE ENGINE (OPERATING ON CHEAP SURROGATE) ---
public array function initPopulation(required numeric popSize, required array bounds) {
    var pop = [];
    for (var i = 1; i <= arguments.popSize; i++) {
        var ind = [];
        for (var d = 1; d <= arrayLen(arguments.bounds); d++) {
            arrayAppend(ind, rand() * (arguments.bounds[d].max - arguments.bounds[d].min) + arguments.bounds[d].min);
        }
        arrayAppend(pop, ind);
    }
    return pop;
}

public array function crossover(required array p1, required array p2) {
    var child = [];
    var alpha = 0.5; // Blend Crossover parameter
    for (var d = 1; d <= arrayLen(arguments.p1); d++) {
        var gamma = (1 + 2 * alpha) * rand() - alpha;
        arrayAppend(child, arguments.p1[d] + gamma * (arguments.p2[d] - arguments.p1[d]));
    }
    return child;
}

public array function mutate(required array individual, required array bounds, numeric rate=0.2, numeric scale=0.5) {
    var mutated = duplicate(arguments.individual);
    for (var d = 1; d <= arrayLen(mutated); d++) {
        if (rand() < arguments.rate) {
            // Box-Muller transform for normal distribution mutation
            var u1 = rand(); var u2 = rand();
            if(u1 == 0) u1 = 0.0001;
            var normalRandom = sqrt(-2.0 * log(u1)) * cos(2.0 * pi() * u2);
            
            mutated[d] += normalRandom * arguments.scale;
            // Clip boundaries
            if (mutated[d] < arguments.bounds[d].min) mutated[d] = arguments.bounds[d].min;
            if (mutated[d] > arguments.bounds[d].max) mutated[d] = arguments.bounds[d].max;
        }
    }
    return mutated;
}

public array function runGaOnSurrogate(required array xTrain, required array yTrain, required array bounds, required numeric popSize, required numeric generations) {
    var population = initPopulation(arguments.popSize, arguments.bounds);
    
    for (var gen = 1; gen <= arguments.generations; gen++) {
        // Evaluate complete generation instantly on the analytical surrogate proxy
        var scoredPop = [];
        for (var i = 1; i <= arrayLen(population); i++) {
            arrayAppend(scoredPop, {
                vector: population[i],
                score: predictSurrogateFitness(population[i], arguments.xTrain, arguments.yTrain)
            });
        }
        
        // Rank-based Sort Descending
        arraySort(scoredPop, function(a, b) {
            return b.score > a.score ? 1 : (b.score < a.score ? -1 : 0);
        });
        
        // Rebuild next generation
        var nextGen = [ scoredPop[1].vector, scoredPop[2].vector ]; // Elitist strategy
        while (arrayLen(nextGen) < arguments.popSize) {
            var parent1 = scoredPop[randRange(1, 5)].vector;
            var parent2 = scoredPop[randRange(1, 5)].vector;
            var child = crossover(parent1, parent2);
            child = mutate(child, arguments.bounds);
            arrayAppend(nextGen, child);
        }
        
        // Update population pointer
        for (var k = 1; k <= arrayLen(nextGen); k++) {
            population[k] = nextGen[k];
        }
    }
    
    // Return best vector from final generation iteration
    return population[1];
}

// --- 4. EXECUTING THE HYBRID SEED LOOP ---
writeOutput("Starting CFML Hybrid GA-RKHS Optimization Engine...<br><br>");

// Step A: Initialize Gram Dataset
variables.X_train = initPopulation(10, variables.bounds);
variables.y_train = [];
for (var i = 1; i <= arrayLen(variables.X_train); i++) {
    arrayAppend(variables.y_train, trueExpensiveFitness(variables.X_train[i]));
}

// Step B: Active Learning Global Intercept Loop
for (var loop = 1; loop <= variables.surrogateMaxIter; loop++) {
    writeOutput("--- Iteration #loop#/#variables.surrogateMaxIter# ---<br>");
    writeOutput("Gram Matrix Dataset Size: " & arrayLen(variables.X_train) & " points<br>");
    
    // Step C: Let the GA scan the math surrogate space aggressively
    var proposedCandidate = runGaOnSurrogate(variables.X_train, variables.y_train, variables.bounds, variables.popSize, variables.generations);
    
    // Step D: Run costly physical simulation check on candidate
    var trueScore = trueExpensiveFitness(proposedCandidate);
    
    writeOutput("GA Proposed Vector: [" & arrayToList(proposedCandidate, ", ") & "]<br>");
    writeOutput("True Real-World Fitness Score: " & numberFormat(trueScore, "0.0000") & "<br>");
    
    // Step E: Augment data matrices
    arrayAppend(variables.X_train, proposedCandidate);
    arrayAppend(variables.y_train, trueScore);
    
    writeOutput("Current System Best Score: " & numberFormat(arrayMax(variables.y_train), "0.0000") & "<br><br>");
}

writeOutput("Optimization Execution Cycle Terminated.");
</cfscript>

Key Architectural Shifts in the CFML Variant

  • Script-Only Architecture: Written purely within <cfscript> blocks to remove legacy tag overhead and maximise processing throughput for vector arithmetic loops.
  • Native Box-Muller Generator: Because ColdFusion’s native rand() produces a flat uniform distribution, a standard Box-Muller transformation was written directly into the mutate() method to correctly inject normal (Gaussian) distribution mutations.
  • Surrogate Mapping Array Matrix: Replaces Python arrays with native array structs. The predictSurrogateFitness method models continuous field behaviours natively via distances weighted dynamically across the current known points.

Lec 50 Kernel Ridge Regression (KRR)

NPTEL - Indian Institute of Science, Bengaluru

YouTube: 09/2025 35:13
By , No comments:
Tags , , , , , ,

In Hilbert Space, All Things Are Quantumly Possible

Mike's Notes

This one got me thinking. John von Neumann's 5 laws might be useful inside Pipi for determining any state. I need to do a lot more reading.

The 5 Quantum Commandments of John von Neumann

    1. An arrow in Hilbert space shall represent the quantum state of any object.
    2. Altering the object shall make the arrow rotate smoothly through Hilbert space to represent a new state.
    3. Distinct axes in Hilbert space shall reflect different possible sets of properties of the object.
    4. The shadow the arrow casts onto an axis shall encode the odds of that possibility being realised when a measurement takes place.
    5. Upon measurement, the arrow shall randomly and instantaneously jump to align with the axis representing the observed outcome, and the object shall acquire a fully determined property.
- Quanta Magazine

The original article has some SVG dynamic diagrams which are missing here.

Resources

References

  • Reference

Repository

  • Home > Ajabbi Research > Library > Subscriptions > Quanta Magazine
  • Home > Handbook > 

Last Updated

02/09/2026

In Hilbert Space, All Things Are Quantumly Possible

By: Charlie Wood
Quanta Magazine: 26/08/2026

Charlie Wood: Staff Writer, Quanta Magazine.

...

To explore quantum phenomena, we must leave the familiar world and enter the abstract realm of Hilbert space.

...

At the heart of quantum mechanics lie a few sacred rules for how to use the theory. First and foremost is, roughly, that thou shalt not think about ordinary objects presently whizzing through ordinary space. Rather, quantum mechanics predicts — in exquisite detail — all the possible ways that an object might turn out to be in the future. Exploring those possible futures requires tracking an entirely different mathematical object — an arrow known as a vector, one oriented in an expansive, alien domain.

These arrows aren’t pointing at locations. “It’s a much more abstract space than that,” said Lucien Hardy, a physicist at the Perimeter Institute for Theoretical Physics in Waterloo, Canada. They’re “really pointing in a direction in a possibility space.”

This possibility space is called Hilbert space, and it acts as the primary arena for quantum physics.

The early quantum pioneers didn’t realize — at first — that the arcane math that strikingly captured the conduct of atoms had left the real world behind. It took a visionary mathematical physicist, John von Neumann, to recognize and define the quantum world as a Hilbert space. Once he did, exploring the ins and outs of Hilbert space would lead physicists to a deeper, more unified understanding of quantum physics.

Here’s how von Neumann’s first commandment of quantum physics came to be, and how to understand it. 

What Is Hilbert Space?

Von Neumann’s commandments, or axioms, were his way of making sense of the two distinct forms of quantum mechanics developed back-to-back in the 1920s. First came Werner Heisenberg’s “matrix mechanics” in 1925. It used inscrutable tables and, in later formulations, interminable towers of numbers to calculate the odds that an electron circling an atom would jump to a higher or lower orbit. The next year, Erwin Schrödinger introduced his “wave mechanics.” It used waves to track, for instance, the probability of a particle being found at a certain location in space. While the pictures evoked by these two physicists looked completely distinct, they yielded identical predictions. Heisenberg and Schrödinger had come up with two radically different incarnations of one theory. But what was that theory?

The question fascinated David Hilbert, a renowned mathematician who had devoted much of his life to rebuilding physics on a sturdy foundation of crisp axioms. He got von Neumann thinking about the problem in the mid-1920s. In 1927 the 23-year-old prodigy — building on insights from Paul Dirac — solved it in a single-author trilogy of papers.

A bearded man sits in a wicker chair.

The mathematician David Hilbert sought a mathematical structure that would unify the different forms of quantum mechanics.

MacTutor

“The ideas specifically were von Neumann’s, but the inspiration — why do you axiomatize and what for — this is something that he took from Hilbert,” said Leo Corry, a historian of mathematics.

These papers laid out the rules for quantum mechanics, carefully defining the theory’s central objects and how they behaved. Von Neumann showed that Heisenberg’s towers and Schrödinger’s waves were reflections of the same entity, just as 0.5 and ½ indicate the same point on the number line. They both represented the main character in quantum mechanics: the quantum state.

Everything has a state. A coin can read heads or tails. A grandfather clock’s bob can take on any number of positions as it swings. Most of physics amounts to capturing an object’s state and predicting how it will change.

Von Neumann’s quantum rules complicate the notion of a state. Before you observe a quantum object, it does not have a fixed set of properties, such as a specific position. Instead it has a combination of possible properties unique to quantum mechanics — a “quantum superposition.” A superposition combines, for instance, all possible places the particle might end up being. Those possibilities can be precise and informative; perhaps there is a 99% chance you’ll find your particle to your left and a 1% chance you’ll find it to your right. You know how to place your bets, but you can’t know for sure if you’ve won until you check.

Von Neumann rendered the quantum state as a mathematical arrow called a vector. This arrow points in some direction through a space capturing all the possible futures of any quantum object — a Hilbert space.

Imagine a quantum traffic light with three possible states — red, yellow, or green. Its arrow exists in a three-dimensional Hilbert space, where the three axes represent the three possible future colors. Until the moment the light is observed, it doesn’t have a color, but rather a mixture of possible colors. So its arrow points into the space’s central region. The more closely the arrow aligns with, say, the red axis, the more likely the light is to shine red.


Diagram

Mark Belan/Quanta Magazine

The state of any object, from an electron to a galaxy, can be captured by such a vector, pointing in some direction through such a Hilbert space. This is von Neumann’s first rule of quantum mechanics. 

What Happens in Hilbert Space?

An arrow moves through Hilbert space in one of two ways. Von Neumann’s other commandments specify how.

The first possibility corresponds to what happens before an observation. As the world influences the object, changing its state, the arrow turns smoothly through Hilbert space. It might get closer to the green axis, which would make our traffic light more likely to be measured as green, or to red or yellow. The point is that all this happens smoothly and predictably.

Then, if you actually observe the system, the vector will instantly and randomly snap onto either the red, yellow, or green axis. The more aligned it is with one axis, the more likely it is to snap to that axis instead of the others, but its fate is ultimately unpredictable. Let’s say it goes green. You’ll observe a green light, and there is now a 100% chance that it will still be green in subsequent measurements, because the arrow is fully aligned with the green axis. The quantum superposition is no more.

Diagram

What Properties Define Hilbert Space? 

The more possible futures an object has, the bigger its Hilbert space. A coinlike particle with two possible futures is a “qubit,” the computational building block of quantum computers. It has a two-dimensional Hilbert space. Our three-color traffic light has a three-dimensional Hilbert space. But that’s just the beginning. A freely floating particle could be found in any location in the universe, so its Hilbert space must span an infinite number of dimensions.

This size — whether it’s two dimensions or an infinite number — is the only fundamental feature of a Hilbert space, according to von Neumann’s rules. The axes are arbitrary and imagined by us; they aren’t intrinsic to the space.

Consider an electron. It has one state, one arrow, pointing in a vast Hilbert space. Its Hilbert space spans all possible measurements — energy, position, momentum, etc. If you are curious where the electron might be, you can mark the space with the axes that represent possible positions. If you are wondering where the particle might be going, you apply a different set of axes, those representing possible momenta. No matter which measurement you intend to make, the underlying Hilbert space remains the same.

This freedom to carve up Hilbert space as we see fit is what allowed Heisenberg and Schrödinger to come up with two distinct versions of the same theory. Heisenberg’s picture essentially put in axes and let them rotate around the vector, while Schrödinger’s picture did the opposite: It put in a fixed set of axes and let the vector rotate relative to them. They were two completely different mathematical perspectives on the same arrows, in the same Hilbert spaces.

A man sits wearing a suit and tie.

John von Neumann developed an underlying structure for quantum mechanics that involves arrows moving inside an abstract Hilbert space.

US Department of Energy

In the first of his 1927 papers, von Neumann laid out two mathematical criteria that defined such a space. First, it had to be “complete.” It couldn’t be missing any regions or points. And second, you had to be able to calculate the alignment between a state and an axis, which you can visualize by imagining a light shining straight down onto an arrow so it casts a shadow on an axis. (The longer the shadow, the more aligned with that axis the arrow is.) The space had to allow for this operation, known as an inner product. Any space with these two features, no matter its size or origin, was a Hilbert space.

“This was a major step in creating what we call Hilbert space quantum mechanics,” said Miklós Rédei, a philosopher of physics at the London School of Economics. “It’s a beautiful example of how mathematical generalization or abstraction takes place.”

Von Neumann referred to these abstract spaces as Hilbert spaces because his mentor Hilbert had been the first mathematician to explore specific spaces with infinite dimensions in the early 1900s. Hilbert’s work had relied on those spaces being complete and having an inner product, but he didn’t think of them as examples of a more general class of spaces until his protégé grouped them together. The older mathematician may have been surprised to find his name attached to this new mathematical structure. “Dr. von Neumann, I am really curious to know what these Hilbert spaces are, after all,” Hilbert reportedly asked during a 1929 lecture. 

Is Hilbert Space Real or Just an Abstraction?

A century after the birth of quantum mechanics, the theory has left physicists in an awkward position. We live in a world where objects change position as they move through three dimensions of physical space. But our most fundamental theory takes place somewhere else, in von Neumann’s vast realm of possibilities. What does that imply about the reality of our world, or that of Hilbert space?

Mathematically Crucial Fine Print

In quantum physics, Hilbert spaces use complex numbers, which involve the imaginary number i, and negative regions. But von Neumann’s fourth commandment guarantees that the odds of any possibility will always come out as a real, positive number.

To Sean Carroll, a philosopher and physicist at Johns Hopkins University, the message is clear. If quantum mechanics is the fundamental theory of nature, then Hilbert space should be considered the fundamental theater of reality, he argued in a 2022 paper(opens a new tab). One of his lines of research seeks to distill our familiar world from the disorienting Hilbert space that encapsulates all the ways the universe could possibly be.

Other physicists take a more pragmatic stance. Jonathan Sorce, a physicist at Princeton University, says that Hilbert space is a handy mathematical construction that is remarkably useful for describing many quantum systems — but not all of them. He belongs to a community of researchers searching for a mathematical construction that can describe the fabric of space and time as a quantum object. Such a theory is a prerequisite for answering big questions such as what goes on at the heart of a black hole.

Physicists asking these questions have recently focused on an even more abstract space that seems especially well suited for their purposes. This kind of space is made up of the things you could do to a Hilbert space, such as slicing it up in different ways or rotating one slice into another. In this arena, they have found that black holes seem a bit less mysterious.

This sort of über-space is known today as a von Neumann algebra. Von Neumann himself helped develop it as a potential remedy for some logical inconsistencies(opens a new tab) with Hilbert space that troubled him. “I would like to make a confession which may seem immoral: I do not believe in Hilbert space anymore,” he wrote in a 1935 letter while exploring the virtues of algebras.

Sorce, for his part, doesn’t share von Neumann’s desire for one space to rule them all. He’s content to use whichever mathematical construction best suits the quantum object he’s studying. Often it’s a Hilbert space. Sometimes it’s a von Neumann algebra. And occasionally it might even be one of the many other spaces mathematicians have cooked up over the last century.

“There’s a whole zoo of these things,” he said.

The 5 Quantum Commandments of John von Neumann

  1. An arrow in Hilbert space shall represent the quantum state of any object.
  2. Altering the object shall make the arrow rotate smoothly through Hilbert space to represent a new state.
  3. Distinct axes in Hilbert space shall reflect different possible sets of properties of the object.
  4. The shadow the arrow casts onto an axis shall encode the odds of that possibility being realised when a measurement takes place.
  5. Upon measurement, the arrow shall randomly and instantaneously jump to align with the axis representing the observed outcome, and the object shall acquire a fully determined property.

By , No comments:
Tags , , ,
Subscribe to: Posts (Atom)