AICrossSim/NewComputeBench
AICrossSim/NewComputeBench is a benchmark suite for new compute paradigms (Spiking neural networks, Optical computation, In-Memory computation, etc) via software emulation. We aim to predict the scaling law of neural networks trained with new compute paradigms by running small & medium scale experiments and extrapolate the trends we observed. NewComputeBench project mainly consists of the following steps:
- Build a scaling framework to support the pre-training of language models up to 1.1B parameters (CLM model series)
- Implement software emulation of new compute paradigms (e.g., optical compute, spiking neural networks, in-memory compute, etc)
- Filter out promising new compute paradigms by running small & medium scale experiments (Roberta on GLUE)
- Scale up the promising new compute paradigms to large-scale language models
- Fine-tuning/pretraining of CLM models (60M - 1.1B)
- Optical compute
- Spiking neural networks
- In-memory compute
- Parameter-efficient fine-tuning of larger LLMs (e.g., Llama-3.1-8B)
What's New
-
🚧4th Oct, 2025 Milestone: Fine-tuning/pretraining of alternative compute paradigms on CLMs.
Item Description Optical Transformer Tutorial -
🚩1th Oct, 2025 Milestone: Fine-tuning/pretraining of alternative compute paradigms on Roberta
Item Description Optical Transformer Tutorial CompleteThis -
🚩 9th, Jun, 2025 Milestone: Our Software-emulation & acceleration backend, Mase-triton, is released on PyPI. Try it via
pip install mase-triton.- For more details, please refer to Intro to Mase-triton and Mase-triton GitHub
-
🚩 15th April, 2025 Milestone: System and model-level training simulation (Small Language Models).
Item Description Environment setup Tutorial Pretraining AICrossSim LLMs (60M, 200M, 400M, 1.1B) & evaluation Tutorial Software-emulated bitflip-aware pretraining & evaluation Tutorial
Roadmap
- Model Training & Evaluation
- LLMs
- Pretraining of LLMs (60M, 200M, 400M, 1.1B) using the Llama-3 architecture.
-
lm-eval-harnessevaluation of LLMs. - Parameter-efficient fine-tuning
- Supervised fine-tuning
- LLMs
- Model Behavior-Level Simulation
- Post-training bitflip transform & bitflip-aware pretraining
- Optical compute
- Roberta
- CLM
- Spiking neural networks
- In-memory compute
About the Project
This project is led by Dr. Yiren Zhao at Imperial College London, Dr. Luo Mai at University of Edinburgh, Prof. Robert Mullins at University of Cambridge, and funded by Advanced Research + Invention Agency (ARIA).