NewComputeBench

NewComputeBench#

NewComputeBench (GitHub) is a benchmark suite for new compute paradigms — Spiking Neural Networks, Optical computation, Processing-in-Memory, and more — via software emulation. The project aims to predict the scaling law of neural networks trained with new compute paradigms by running small- and medium-scale experiments and extrapolating observed trends.

The project is led by Dr. Yiren Zhao (Imperial College London), Dr. Luo Mai (University of Edinburgh), and Prof. Robert Mullins (University of Cambridge), and is funded by the Advanced Research + Invention Agency (ARIA).

Project Overview#

NewComputeBench is structured around three phases:

  1. Build a scaling framework to support pretraining of language models up to 1.1B parameters (the AICrossSim-CLM series).

  2. Implement software emulation of new compute paradigms (optical compute, spiking neural networks, in-memory compute, etc.).

  3. Filter out promising paradigms through small- and medium-scale experiments, then scale up.

Current status:

  • ✅ Scaling framework for CLM pretraining (60M – 1.1B)

  • ✅ Software emulation of Random Bitflip, Optical Compute, Spiking Neural Networks, PIM

  • ✅ RoBERTa experiments on GLUE (sanity checks)

  • ✅ CLM bitflip-aware pretraining and LoRA fine-tuning of Llama-3.1-8B

  • ✅ Scaling bitflip-aware training to 70B parameter-efficient fine-tuning (Llama-3-70B, FSDP2)

Roadmap#

Model Training & Evaluation

  • ✅ Pretraining of CLM models (60M, 200M, 400M, 1.1B) using the Llama-3 architecture

  • ✅ lm-eval-harness evaluation of pretrained CLMs

  • ✅ Parameter-efficient fine-tuning (LoRA)

Model Behaviour-Level Simulation

  • ✅ Random Bitflip

    • ✅ Post-training bitflip transform

    • ✅ Bitflip-aware pretraining (60M – 1.1B)

    • ✅ Bitflip-aware LoRA fine-tuning (Llama-3.1-8B)

    • ✅ Bitflip-aware LoRA fine-tuning at scale (Llama-3-70B, FSDP2)

  • ✅ Optical Compute

    • ✅ RoBERTa fine-tuning (125M)

    • ✅ CLM full fine-tuning (60M – 1.1B)

    • ✅ CLM parameter-efficient fine-tuning (60M – 1.1B)

  • ✅ Spiking Neural Networks

    • ✅ RoBERTa fine-tuning (125M)

  • ✅ Processing in Memory

    • ✅ RoBERTa fine-tuning (125M)

    • ✅ ViT-Base fine-tuning (86M)

What’s New#

31 May 2026 — Scaling bitflip-aware training to 70B parameter-efficient fine-tuning. Bitflip-only eval degrades Llama-3-70B perplexity 6.17 → 89.28; bitflip-aware LoRA (r = 32, lr = 2e-4) under FSDP2 + torchtitan converges training loss to ~2.5 – 2.7 and was early-stopped at 7,900 steps, demonstrating that the 8B recipe transfers cleanly to 70B. See Bitflip-Aware LoRA Fine-Tuning with FSDP2 (Llama-3-70B).

4 Feb 2026 — Bitflip-aware LoRA fine-tuning of Llama-3.1-8B. LoRA adapters with only 1.2% trainable parameters reduce perplexity from 1008.95 to 11.01 (clean baseline: 7.91). See Bitflip-Aware LoRA Fine-Tuning.

4 Oct 2025 — Optical Transformer fine-tuning on CLM models (60M – 1.1B). See Scaling Optical Transformers to Causal Language Models.

1 Oct 2025 — Optical Transformer, Spiking Transformer, and PIM experiments on RoBERTa. See Optical Neural Networks on RoBERTa, Spiking Neural Networks on RoBERTa, Processing-in-Memory on RoBERTa.

9 Jun 2025 — Mase-triton released on PyPI (pip install mase-triton). See Mase-Triton.

15 Apr 2025 — System and model-level training simulation (Small Language Models). Environment setup, pretraining of AICrossSim-CLM (60M – 1.1B), and bitflip-aware pretraining. See Installation and LLM Pretraining & Evaluation.