How to recompile models for the AIPUs?

From what I understand here running custom models is still experimental and requires a lot of steps

https://github.com/axelera-ai-hub/voyager-sdk/blob/release/v1.5/docs/tutorials/custom_model.md

Isn’t it possible to come up with some kind of general translation layer to run models that are compiled for GPUs? 

Well I’m too interested on different models like codellama and others

​@Spanner  any tips for this behavior ?

Is this the secret???

https://github.com/axelera-ai-hub/voyager-sdk/blob/release/v1.5/docs/reference/compiler_cli.md

or this???

https://github.com/axelera-ai-hub/voyager-sdk/blob/release/v1.5/docs/tutorials/llm.md#supported-pipelines

​@florit Thanks, those resources seems to be what I was looking for. Sounds like recompilation is quite easy. Not sure how easy it is to convince the ollama team to recompile and host alle their models for this architecture. They’d probably do it if the performance is good enough. Can’t wait to see benchmarks for the Europa chip with common LLMs!

​@jclsn only for the 4x4 64GB pcie Card i See a Little Chance to be good enough for the ollama Team.

Yes, me too. The 4x4 64GB will probably cost over 1k though! It would be able to fit huge 70b models, but I doubt it could run them performantly with around 30 tokens/s. I talked to some Ollama guys on Discord. They think the cards are mainly made for object detection models. They might have a point. 
If I am wrong and it indeed could run those large models performantly, I would be interested in buying one to have a local low-power chatbot. Useful LLMs start at 32b-70b. Everything below is not worth the money I found when playing around with 700€ GPUs with 16GB VRAM.

The cost per Card is in my Country arround 2k€

on the 4x4 64GB

on my MacBook I got 64GB unified ram and 32 GPU cores

means 24GB GPU with 32 cores

this card is a huge benefit for my mini pc with 64GB VRAM and 16 aipu cores

because I can load larger models like “r1-1776:70B”

and this would be cool for an offline model

at the moment the mini pc can load the Ollama r1-1776:70B model but processing slow with the 80 TOPS and 128GB RAM he got

but it works … slowly but it works

can’t wait for a 70B model ​@Spanner this I want !!!

Are there prices for the Europa published already? Maffeo said under 1000$ in the talk here

That 16GB Card Starts at ~1200€ without tax

and yes that under 1000$ without tax

nothing wrong … so you Must add ~1700€ +350€ tax + delivery

so in my case this is ~2100€ for the 64GB Card 

Look at this link there you Found a co-GPU with 64GB for Servers 

https://shop.prostor.de/de/pny-nvidia-a16-64gb.html

this is what he Talking about 

Europa pricing hasn’t been released yet, but we’ll make sure it’s announced on here as soon as there’s any news 👍

but what’s about pricing in Europe?

I’ve ordered a card yet and price are very high for the 64GB Version of the Metis 4x4 Version…

​@Spanner  any updates we hasn’t heard before you?

Is this what you’re looking for, ​@florit? The 4 Quad-Core PCIe card?  It’s got the pricing on the store page, in Euros 👍

yes ​@Spanner 

and when is it out for delivery?

I’ve got no updates … only receive a bill

Oh I see! Sorry, I thought you meant you were looking to buy one 😅 But you’ve already got one on order, and you’re looking for a delivery date, is that right?

Let me ask around the team 👍

I’ll DM you as soon as I get some info, by the way 🤙