AICrossSim/NewComputeBench
AICrossSim/NewComputeBench is a benchmark suite for new compute paradigms (Spiking neural networks, Optical computation, In-Memory computation, etc) via software emulation. We aim to predict the scaling law of neural networks trained with new compute paradigms by running small & medium scale experiments and extrapolate the trends we observed. NewComputeBench project mainly consists of the following steps:
- Build a scaling framework to support the pre-training of language models up to 1.1B parameters (CLM model series)
- Implement software emulation of new compute paradigms (e.g., optical compute, spiking neural networks, in-memory compute, etc)
- Filter out promising new compute paradigms by running small & medium scale experiments (Roberta on GLUE)
- Scale up the promising new compute paradigms to large-scale language models
- Fine-tuning/pretraining of CLM models (60M - 1.1B)
- Random bitflip
- Optical compute
- Spiking neural networks
- Parameter-efficient fine-tuning of larger LLMs (e.g., Llama-3.1-8B)
- Random bitflip (promising results)
- Optical compute (failed to converge)
What's New
-
4th, Feb, 2026 Milestone: We have successfully fine-tuned Llama-3.1-8B with random bitflip noise injected in forward passes, and observed promising results that the LoRA adapters with only 1.2% trainable parameters can effectively mitigate the effect of noise (reducing perplexity from 1008.95 to 11.01, with the original clean perplexity at 7.91).
Item Description Llama-3.1-8B with random bitflip noise Tutorial -
4th Oct, 2025 Milestone: Fine-tuning/pretraining of alternative compute paradigms on CLMs.
Item Description Optical Transformer Tutorial -
🚩1th Oct, 2025 Milestone: Fine-tuning/pretraining of alternative compute paradigms on Roberta
Item Description Optical Transformer Tutorial Spiking Transformer Tutorial Processing in Memory Tutorial -
🚩 9th, Jun, 2025 Milestone: Our Software-emulation & acceleration backend, Mase-triton, is released on PyPI. Try it via
pip install mase-triton.- For more details, please refer to Intro to Mase-triton and Mase-triton GitHub
-
🚩 15th April, 2025 Milestone: System and model-level training simulation (Small Language Models).
Item Description Environment setup Tutorial Pretraining AICrossSim LLMs (60M, 200M, 400M, 1.1B) & evaluation Tutorial Software-emulated bitflip-aware pretraining & evaluation Tutorial
Roadmap
- Model Training & Evaluation
- LLMs
- Pretraining of LLMs (60M, 200M, 400M, 1.1B) using the Llama-3 architecture.
-
lm-eval-harnessevaluation of LLMs. - Parameter-efficient fine-tuning
- LLMs
- Model Behavior-Level Simulation
- Lossy Communication (random bitflip)
- Post-training bitflip transform
- Bitflip-aware pretraining (60M - 1.1B)
- Bitflip-aware parameter-efficient fine-tuning (Llama-3.1-8B)
- Optical compute
- Roberta Fine-tuning (125M)
- CLM full fine-tuning (60M - 1.1B)
- CLM parameter efficient fine-tuning (60M - 1.1B)
- Spiking neural networks
- Roberta Fine-tuning (125M)
- Processing in Memory
- Roberta Fine-tuning (125M)
- ViT-Base Fine-tuning (86M)
- Lossy Communication (random bitflip)
About the Project
This project is led by Dr. Yiren Zhao at Imperial College London, Dr. Luo Mai at University of Edinburgh, Prof. Robert Mullins at University of Cambridge, and funded by Advanced Research + Invention Agency (ARIA).