AICrossSim/NewComputeBench
AICrossSim/NewComputeBench is a benchmark suite for new compute paradigms (Spiking neural networks, Optical computation, In-Memory computation, etc) via software emulation. We aim to predict the scaling law of neural networks trained with new compute paradigms by running small & medium scale experiments and extrapolate the trends we observed. NewComputeBench project mainly consists of three parts:
- Model Training
- Model Behavior-Level Simulation
- Hardware-Performance Simulation (
🚧 TODO)
What's New
-
🚩 15th April Milestone: System and model-level training simulation (Small Language Models).
Item Description Environment setup Tutorial Pretraining AICrossSim LLMs (60M, 200M, 400M, 1.1B) & evaluation Tutorial Software-emulated bitflip-aware pretraining & evaluation Tutorial
Roadmap
- Model Training & Evaluation
- LLMs
- Pretraining of LLMs (60M, 200M, 400M, 1.1B) using the Llama-3 architecture.
-
lm-eval-harnessevaluation of LLMs. - Parameter-efficient fine-tuning
- Supervised fine-tuning
- LLMs
- Model Behavior-Level Simulation
- Post-training bitflip transform & bitflip-aware pretraining
- Optical compute
- Spiking neural networks
- In-memory compute
- Hardware-Performance Simulation
- Hardware performance prediction
🚧 TODO
About the Project
This project is led by Dr. Yiren Zhao at Imperial College London, Dr. Luo Mai at University of Edinburgh, Prof. Robert Mullins at University of Cambridge, and funded by Advanced Research + Invention Agency (ARIA).