Hello everyone,
We are benchmarking some neural network models on the Axelera Metis card and comparing performance with regular PyTorch. We have compiled and quantised the models and can run them on the Metis AIPU.
To test efficiency, we’ve created a ‘model’ consisting of one big matrix multiplication. It requires about 10.9 MFLOPs per inference (measured in PyTorch) and takes about 0.516 ms per inference on the card (on-device, batch size 1, 4 cores, input shape: (1, 57, 1, 1)). That corresponds to roughly 0.021 TFLOPS, about four orders of magnitude below the documented peak of 214 TOPS.
It could be that this model is too small to saturate the card and that overhead dominates. We’ve tried to simply increase the size of this test model further, but that no longer fits in the card’s on-chip weight memory. Furthermore, trying to compile a model with batch size > 1 returns “AssertionError: Only batch sizes of 1 supported”.
We add a convolution layer to the input tensor (4D) to make it compatible with the inspected input shape (2D) of the original model. We currently initialize it with Dirac weights to ensure it behaves like the identity operator.
What is the recommended way to get higher utilisation in this situation? For example, is there a supported way to batch with this kind of model?
We’re happy to share the model architecture or logs if helpful.
Thanks in advance!
