Mild disclosure up front: there's a sale on, and this is me telling you about it. 😄I don’t normally get too commercial in here, but the memory ceiling is one of the questions that comes up quite a lot, and the Axelera Edge 130p in its 16GB configuration is a pretty simple answer to that.
Two things it actually buys you, as opposed to just "more is better":
- Bigger models load. The small LLMs are fine on 4GB. Llama 1B and 3B, no drama. Once you go past that it’s not much slow, as just doesn't fit, and there's not really any way round that.
- Model switching stops being a problem. If you're juggling four or more models and swapping between them, they all need to stay resident. 4GB runs out once a couple of those models are on the larger side. This is the one I'd underline, because the multi-model and multi-stream threads in here are pretty common.
I think it’s actually a strong approach, prototyping on a nice, big 16GB card so you're not fighting memory limits while the pipeline is still under development. Then once it's all locked in, you can figure out the minimum device your specific project will actuall run on.
So, the sales bit: 10% off, €635,95 down from €706,95, or $689.95 from $766.95. Runs until 15 September.
https://store.axelera.ai/products/axelera-edge-130p-16gb
Question for the room: what are you actually hitting the ceiling on? I'd like to know whether it's one big model that won't fit, or a pile of medium ones that won't co-exist, because those are quite different problems and I'm not sure which is the more common one here. If you've already got a 16GB card, does the model-switching thing hold up in practice
