NewComputeBench

NewComputeBench#

NewComputeBench

NewComputeBench (GitHub) is a benchmark suite for new compute paradigms — Spiking Neural Networks, Optical computation, Processing-in-Memory, and more — via software emulation. The project aims to predict the scaling law of neural networks trained with new compute paradigms by running small- and medium-scale experiments and extrapolating observed trends.

The project is led by Dr. Yiren Zhao (Imperial College London), Dr. Luo Mai (University of Edinburgh), and Prof. Robert Mullins (University of Cambridge), and is funded by the Advanced Research + Invention Agency (ARIA).

Project Overview#

NewComputeBench is structured around three phases:

  1. Build a scaling framework to support pretraining of language models up to 1.1B parameters (the AICrossSim-CLM series).

  2. Implement software emulation of new compute paradigms (optical compute, spiking neural networks, in-memory compute, etc.).

  3. Filter out promising paradigms through small- and medium-scale experiments, then scale up.

Current status:

  • ✅ Scaling framework for CLM pretraining (60M – 1.1B)

  • ✅ Software emulation of Random Bitflip, Optical Compute, Spiking Neural Networks, PIM

  • ✅ RoBERTa experiments on GLUE (sanity checks)

  • ✅ CLM bitflip-aware pretraining and LoRA fine-tuning of Llama-3.1-8B

  • ⏹️ Full CLM scaling for Optical Compute, SNN, PIM

Roadmap#

Model Training & Evaluation

  • ✅ Pretraining of CLM models (60M, 200M, 400M, 1.1B) using the Llama-3 architecture

  • ✅ lm-eval-harness evaluation of pretrained CLMs

  • ✅ Parameter-efficient fine-tuning (LoRA)

Model Behaviour-Level Simulation

  • ✅ Random Bitflip

    • ✅ Post-training bitflip transform

    • ✅ Bitflip-aware pretraining (60M – 1.1B)

    • ✅ Bitflip-aware LoRA fine-tuning (Llama-3.1-8B)

  • ✅ Optical Compute

    • ✅ RoBERTa fine-tuning (125M)

    • ✅ CLM full fine-tuning (60M – 1.1B)

    • ✅ CLM parameter-efficient fine-tuning (60M – 1.1B)

  • ✅ Spiking Neural Networks

    • ✅ RoBERTa fine-tuning (125M)

  • ✅ Processing in Memory

    • ✅ RoBERTa fine-tuning (125M)

    • ✅ ViT-Base fine-tuning (86M)

What’s New#

4 Feb 2026 — Bitflip-aware LoRA fine-tuning of Llama-3.1-8B. LoRA adapters with only 1.2% trainable parameters reduce perplexity from 1008.95 to 11.01 (clean baseline: 7.91). See Bitflip-Aware LoRA Fine-Tuning.

4 Oct 2025 — Optical Transformer fine-tuning on CLM models (60M – 1.1B). See Scaling Optical Transformers to Causal Language Models.

1 Oct 2025 — Optical Transformer, Spiking Transformer, and PIM experiments on RoBERTa. See Optical Neural Networks on RoBERTa, Spiking Neural Networks on RoBERTa, Processing-in-Memory on RoBERTa.

9 Jun 2025 — Mase-triton released on PyPI (pip install mase-triton). See Mase-Triton.

15 Apr 2025 — System and model-level training simulation (Small Language Models). Environment setup, pretraining of AICrossSim-CLM (60M – 1.1B), and bitflip-aware pretraining. See Installation and LLM Pretraining & Evaluation.