Hi ​@llrds ,

Where I am in the process is that I am trying to figure out how to integrate the Axelera inferencing into my python/opencv application. The example applications in the sdk mostly work with streams, which I understand is the best way to run very fast. however, in my application I need more control over the camera (frame size, frame rate, etc...normally supplied as arguments to gstreamer or v4l2) . I also want to do my own visualizations on the images using opencv. However, this is where things seems not so straight foward, at-least to someone with (limited) skills like me :) . I have explored, and continue to explore several directions:

• There is Axinference in the sdk that is supposed to do inference on images from opencv, but the example is for classification, while my use-case is for detection → I still need to go through it in detail, understand it, and then modify it for detection use case. I expect this will take some time.
• I thought I’d take the application.py example and use the inference results meta data and images returned and then do my own visualizations and processing of data. However, I struggled to convert the images to an openCV format (I tried reading through the code in the sdk to figure out but so far I haven’t succeeded). 
• I also saw that the sdk should allow adding inferencing as gstreamer elements, and I was thinking this could be one way to interface between my python/opencv application and the inferencing part (using appsink elements in gstreamer). However, the opencv included in the voyager-sdk is built without gstreamer, so now I am trying to do a custom build with gstreamer. I saw an example to run inferencing as a gstreamer pipeline, but I couldn’t get that to work somehow,...once I have my gstreamer enabled in opencv, then I will re-attempt that.

So this is where I am right now. I will continue trying to get something to work. So far it’s a struggle Ideally, if there was a simple inferencing function “doInferencing(image)”, to which I could give and image and get the inference results back, would have been ideal (even if not the best way to get the fastest inferencing speed on a stream).

Anyway, this is part of the learning curve I guess. I will continue exploring. 

Pls do share if you find anything useful!

Hi ​@saadtiwana 

At DeGirum (a SW partner of Axelera), we developed PySDK which does exactly what you want (because we built it for developers like you ). Our team is working on developing detailed user guides specific to Axelera, but in the meantime you can see a starting example at: PySDKExamples/examples/google_colab/pysdk_axelera_hello_world.ipynb at main · DeGirum/PySDKExamples. You can find a lot of other usage examples at: DeGirum. 

We will be happy to help you with your use case (and hop on a call if you need live help). We have helped a lot of developers with other hardware, so we are very familiar with the challenges developers face when trying to write their own logic on top of APIs provided by the HW vendor. Please take a look and see if you find it useful.

Hi ​@shashi.chilappagari, Thank you for your message.

I actually forgot to mention in my post that I did look at Degirum also few days ago, but, in my limited time spent, struggled to find an example detailing how to run a model on the Axelera hardware (most examples used the cloud inference). I somehow didn’t see the example that you just linked..this seems to have a section where it runs inference on the Axelera hardware too. I will try it out tonight and get back to you if any issues. This sure does seem to do what I need.

Btw, the models needed for inference on Axelera hardware using Degirum, are they exactly same as the output of the axelera compiler or they must go through your online compiler? In other words, can I take a model from the Axelera model zoo and use it as-is with Degirum?

Hi ​@saadtiwana 

The Axelera integration is relatively new but internally we have had multiple models and use cases working. 

Yes, you can take a model from Axelera model zoo and use it with DeGirum PySDK, but you need to configure it properly. This is where our user guides come in handy. If there are any models at the top of your list, please do let us know. We will integrate them so that they will work for you out-of-the-box. 

Hi ​@shashi.chilappagari 

My first and main usage is for detection models. I did not yet get to the point of selecting the exact model yet, but it will have to be one that I can retrain myself easily on custom dataset later, so I would think one of the yolo models from ultralytics would be the one. I would like to first get any detection model running on the Axelera hardware, after which I can narrow down to a specific model. 

Btw, I just looked through the tutorial you shared. I see what while it’s running the model on Axelera hardware, the hardware itself is still running on your cloud servers. Do you have an example where I can run a simple object detection on my local Axelera hardware instead? I am using the Aetina RK3588 board with the Metis M.2, which runs Ubuntu 22 as OS. Actually this is where I had issues the last time I was looking at DeGirum tutorials because most of them were running the models on the cloud. If you can point me to an example I can run on my hardware, that would be an excellent start. Also, do you have any public version of the Axelera user guide you mention?

Thanks for replying ​@saadtiwana, I also had a Python/OpenCV app that I created which is working as a PoC on my laptop, like you I’m just trying to make that link then I should be good to go. 

I saw in the application tutorial you can output a numpy array ( frame_result.image.asarray() ), that will be my first port of call as I don’t think I need to dip into Gstreamer. So far I get strange DMX errors when running examples so I need to get over that hurdle before I play with SDK output

​@llrds thanks for pointing me to it, it seems that I missed that very important detail I will try it out tonight. 

Regarding the issues you are seeing, I have never seen those. Do you think doing a reimage of your board and starting over will help? I spent quite some time doing it past weeks due to running into storage issues, and wrote down the full procedure in github to avoid spending the time again. You can refer to it here:

SaadTiwana/axelera: Notes and files for work on Axelera Metix M.2 on RK3588 board

Look at the file under “board preparation” folder. The whole process should not take too long if you follow along. I think this might be the best way to start clean :)

Hi ​@saadtiwana 

The main advantage of PySDK is that once you have the hardware setup locally, the code developed for cloud inference can run locally by changing @cloud to @local. This allows people to start/continue development without having to battle hardware setup issues. Once you install PySDK (pip install degirum), please type

degirum sys-info

 If you see Axelera Metis listed as one of the devices, you are good to go. FYI, we are using Axelera runtime 1.3.1. If this is installed on your system, PySDK will pick it up. Please let us know if you encounter any issues. 

Hi ​@saadtiwana 

My first and main usage is for detection models. I did not yet get to the point of selecting the exact model yet, but it will have to be one that I can retrain myself easily on custom dataset later, so I would think one of the yolo models from ultralytics would be the one. I would like to first get any detection model running on the Axelera hardware, after which I can narrow down to a specific model. 

Perfect. We already have multiple detection models in our model zoo (coco, license plate, face, fire&smoke, ppe, hand, person). Our cloud compiler supports taking custom YOLO checkpoints and compiling for Axelera. So your use case should be fully supported. 

 If you can point me to an example I can run on my hardware, that would be an excellent start. Also, do you have any public version of the Axelera user guide you mention?

The guides are not public yet. Once we are done, we will share with the community. In the meantime, if you install using the instructions at Overview | DeGirum Docs, you should be able to run the examples that work on cloud locally also by changing @cloud to @local.

degirum sys-info

Hi ​@shashi.chilappagari , I ran into some issues. I have followed the installation instructions for linux, and also followed the driver installation instructions for Axelera here since degirum sys-info did not show Axelera device on the machine. I noticed that one package had to be downgraded (metis-dkms 1.0.2 - was using 1.0.3 before). Note that I had previously installed the latest Voyager SDK to my machine.

Anyway, after going through the process in the driver installation page without any errors and hard-power-cycling the board, I still don’t see the Metis device.

Moreover, now even the Axelera sdk has issues refreshing the device as below:

I have a feeling this may be due to the older version of runtime used by the Degirum (that we installed during driver installation?)

Hi ​@saadtiwana 

Thanks for sharing this. I will loop in our SW engineer who can help with this issue. One nice thing I see from your logs is that the Rockchip runtime RKNN is picked up by PySDK. This will allow you to use same codebase to run some models on the built-in NPU of the RK3588 (which has 6TOPS and can run detection models at >60FPS).

Btw, if i switch back to metis-dkms=1.0.3, I can use the hardware with voyager-sdk again, but still does not appear in degirum sys-info . For your info, my M.2 card is running the latest firmware, from the latet voyager-sdk release:

Yes, you are right, and I noticed that too earlier. In fact, i believe they are appearing because I had previously installed the Rockchip drivers too as per your driver installation guide :) It’s something I will try as well. That said, I do need to get the models running on the Axelera hardware anyway, so help in that regard will be appreciated. Ideally, if degirum works with the latest Axelera runtime, that will be ideal so I don’t need to switch between things continuously Thanks.

Hi ​@saadtiwana 

We will definitely focus on getting it to work locally for you. In the initial stages, it is quite common to have versions not in sync, but we are working with Axelera to make sure PySDK releases stay in sync with Axelera updates. Good thing is that we have the same system as you, so we might bother you with same questions regarding the exact setup you have. Once we get it work internally with the version you have, we will make a new release. Our hope is that we get things to work soon so that you don’t have to switch. Meanwhile, please feel free to browse pysdkexamples as the code for cloud and local is same except for 1 line.

​@shashi.chilappagari Thanks, I was able to get the code working on my board with the RKNN backend and getting a very decent 25-30fps with the “yolov5s_relu6_coco--640x640_quant_rknn_rk3588_1” model. I did notice a fairly high CPU usage though  (~50% across cores) via htop, which maybe due to the pre/post processing? not sure.

Anyway, this is still quite amazing how easy it was to get it all running. Next I am looking forward to getting some models running using the Axelera backend locally. Pls do let me know when there’s an update on that. Thanks.

Hi ​@saadtiwana 

Glad to hear that you got PySDK working locally on RKNN. It should give you an idea of how to use the APIs. Yes, the high CPU usage is due to pre and post processing (mainly decoding and resizing). Anyways, there are a lot of optimizations that are relatively easy to do (like the ones using gstreamer that you have been trying). Once you get things working, we can guide you on optimizing the performance. 

Regarding Axelera: we replicated your issue and confirmed that it is due to PySDK supporting runtime 1.3.1 and your system having runtime 1.3.3. Internally, we were able to build a new PySDK that supports 1.3.3 and it passed all tests. An official release of PySDK with support for 1.3.3 will be made mid next week. I will keep you posted.