Custom Model on Metis SBC

Hi there ​@spectral369! Welcome to the show, and thanks for the detailed write-up. You've clearly put a lot of work into the export already, nice one. 👍

So,for LLM models on Metis the supported path at present (I beleive) is the precompiled AxLLM flow rather than generic axcompile. The LLM tutorial has a bit deeper info on this. All the SLMs on Metis are precompiled, and arbitrary models can't be loaded unless they've been compiled for the platform (catch 22, at the moment anyway -- this is getting a lot of work right now though!). I bet the QTools error you're hitting is the compiler saying it doesn't have rules for some of the Constant ops. Which in turn fits with Transformer-class embedding models like Granite not being on the generic ONNX compile path at the moment. 

Before going further with the export, maybe it’d be illuminating to try running one of the precompiled SLMs as a sanity check? Something like axllm llama-3-2-1b-1024-4core-static --prompt "Hello" should run on the MCB. The walkthrough is in that same LLM tutorial. If that runs cleanly, we know your Computer Board and SDK install are good and you've seen the supported LLM path working. If it errors, we can dig in deeper from there.

Out of curiosity, what's the wider use case?

Hello,

Thank you for your swift response!

I tried the SBC tutorial, and it worked as described. At first, I was a bit confused by the stripped-down OS, just Docker with no additional packages but I figured it out after manually installing a few from the Debian repositories 😅. The Llama model worked, and that was the extent of it, since I had to build my demo app on my PC and so far haven’t had a reason to push it to the SBC.

The demo use case is both …..naive and complex. At its core, it’s a custom RAG system using Docling and Qdrant for sensitive data. Since Granite doesn’t support my native language, I’m also using [Helsinki-NLP](https://huggingface.co/Helsinki-NLP). For now, the data ingestion is at a ~decent level. I could elaborate, but it would take a few paragraphs.

Is there an experimental workflow for my case, so I can test it myself ?
Is there anything I can do besides waiting for llm precompilation ?

Thank you,
Peter

That’s not naive at all, it’s a great use case! Really interested to see how that progresses.

So, the custom model deployment tutorial we currently have is very much focused on vision models, since that’s primarily Axelera’s bread and butter. It doesn’t currently have an LLM angle to it, so unfortunately wouldn’t get you closer to Granite.

At this stage, probably the best thing you can do is add it as a feature request in the Launchpad section. The team does keep a close eye on that when working on priorities and the roadmap, so if it’s in there, it’ll get seen. And I’ll also pass your request along internally (we have a group chat specifically about new models people are looking for) to give it a bump there, too.

So, is it Finnish you’re looking to use? (Guessing from the model name!) . I’m just wondering if it's one of the ones Llama 3.2 supports there might be a path that drops the Helsinki-NLP translation step out of your stack entirely? Not a like-for-like Granite replacement, but maybe worth considering depending on your accuracy requirements.