Life after application.py - Iterative testing and learning

Hi ​@llrds ,

Where I am in the process is that I am trying to figure out how to integrate the Axelera inferencing into my python/opencv application. The example applications in the sdk mostly work with streams, which I understand is the best way to run very fast. however, in my application I need more control over the camera (frame size, frame rate, etc...normally supplied as arguments to gstreamer or v4l2) . I also want to do my own visualizations on the images using opencv. However, this is where things seems not so straight foward, at-least to someone with (limited) skills like me :) . I have explored, and continue to explore several directions:

• There is Axinference in the sdk that is supposed to do inference on images from opencv, but the example is for classification, while my use-case is for detection → I still need to go through it in detail, understand it, and then modify it for detection use case. I expect this will take some time.
• I thought I’d take the application.py example and use the inference results meta data and images returned and then do my own visualizations and processing of data. However, I struggled to convert the images to an openCV format (I tried reading through the code in the sdk to figure out but so far I haven’t succeeded). 
• I also saw that the sdk should allow adding inferencing as gstreamer elements, and I was thinking this could be one way to interface between my python/opencv application and the inferencing part (using appsink elements in gstreamer). However, the opencv included in the voyager-sdk is built without gstreamer, so now I am trying to do a custom build with gstreamer. I saw an example to run inferencing as a gstreamer pipeline, but I couldn’t get that to work somehow,...once I have my gstreamer enabled in opencv, then I will re-attempt that.

So this is where I am right now. I will continue trying to get something to work. So far it’s a struggle :) Ideally, if there was a simple inferencing function “doInferencing(image)”, to which I could give and image and get the inference results back, would have been ideal (even if not the best way to get the fastest inferencing speed on a stream).

Anyway, this is part of the learning curve I guess. I will continue exploring. 

Pls do share if you find anything useful!

Hi ​@saadtiwana 

At DeGirum (a SW partner of Axelera), we developed PySDK which does exactly what you want (because we built it for developers like you 😀). Our team is working on developing detailed user guides specific to Axelera, but in the meantime you can see a starting example at: PySDKExamples/examples/google_colab/pysdk_axelera_hello_world.ipynb at main · DeGirum/PySDKExamples. You can find a lot of other usage examples at: DeGirum. 

We will be happy to help you with your use case (and hop on a call if you need live help). We have helped a lot of developers with other hardware, so we are very familiar with the challenges developers face when trying to write their own logic on top of APIs provided by the HW vendor. Please take a look and see if you find it useful.

Hi ​@shashi.chilappagari, Thank you for your message.

I actually forgot to mention in my post that I did look at Degirum also few days ago, but, in my limited time spent, struggled to find an example detailing how to run a model on the Axelera hardware (most examples used the cloud inference). I somehow didn’t see the example that you just linked..this seems to have a section where it runs inference on the Axelera hardware too. I will try it out tonight and get back to you if any issues. This sure does seem to do what I need.

Btw, the models needed for inference on Axelera hardware using Degirum, are they exactly same as the output of the axelera compiler or they must go through your online compiler? In other words, can I take a model from the Axelera model zoo and use it as-is with Degirum?

Hi ​@saadtiwana 

The Axelera integration is relatively new but internally we have had multiple models and use cases working. 

Yes, you can take a model from Axelera model zoo and use it with DeGirum PySDK, but you need to configure it properly. This is where our user guides come in handy. If there are any models at the top of your list, please do let us know. We will integrate them so that they will work for you out-of-the-box. 

Hi ​@shashi.chilappagari 

My first and main usage is for detection models. I did not yet get to the point of selecting the exact model yet, but it will have to be one that I can retrain myself easily on custom dataset later, so I would think one of the yolo models from ultralytics would be the one. I would like to first get any detection model running on the Axelera hardware, after which I can narrow down to a specific model. 

Btw, I just looked through the tutorial you shared. I see what while it’s running the model on Axelera hardware, the hardware itself is still running on your cloud servers. Do you have an example where I can run a simple object detection on my local Axelera hardware instead? I am using the Aetina RK3588 board with the Metis M.2, which runs Ubuntu 22 as OS. Actually this is where I had issues the last time I was looking at DeGirum tutorials because most of them were running the models on the cloud. If you can point me to an example I can run on my hardware, that would be an excellent start. Also, do you have any public version of the Axelera user guide you mention?

Thanks for replying ​@saadtiwana, I also had a Python/OpenCV app that I created which is working as a PoC on my laptop, like you I’m just trying to make that link then I should be good to go. 

I saw in the application tutorial you can output a numpy array ( frame_result.image.asarray() ), that will be my first port of call as I don’t think I need to dip into Gstreamer. So far I get strange DMX errors when running examples so I need to get over that hurdle before I play with SDK output

​@llrds thanks for pointing me to it, it seems that I missed that very important detail :) I will try it out tonight. 

Regarding the issues you are seeing, I have never seen those. Do you think doing a reimage of your board and starting over will help? I spent quite some time doing it past weeks due to running into storage issues, and wrote down the full procedure in github to avoid spending the time again. You can refer to it here:

SaadTiwana/axelera: Notes and files for work on Axelera Metix M.2 on RK3588 board

Look at the file under “board preparation” folder. The whole process should not take too long if you follow along. I think this might be the best way to start clean :)

Hi ​@saadtiwana 

The main advantage of PySDK is that once you have the hardware setup locally, the code developed for cloud inference can run locally by changing @cloud to @local. This allows people to start/continue development without having to battle hardware setup issues. Once you install PySDK (pip install degirum), please type

degirum sys-info

 If you see Axelera Metis listed as one of the devices, you are good to go. FYI, we are using Axelera runtime 1.3.1. If this is installed on your system, PySDK will pick it up. Please let us know if you encounter any issues. 

Hi ​@saadtiwana 

My first and main usage is for detection models. I did not yet get to the point of selecting the exact model yet, but it will have to be one that I can retrain myself easily on custom dataset later, so I would think one of the yolo models from ultralytics would be the one. I would like to first get any detection model running on the Axelera hardware, after which I can narrow down to a specific model. 

Perfect. We already have multiple detection models in our model zoo (coco, license plate, face, fire&smoke, ppe, hand, person). Our cloud compiler supports taking custom YOLO checkpoints and compiling for Axelera. So your use case should be fully supported. 

 If you can point me to an example I can run on my hardware, that would be an excellent start. Also, do you have any public version of the Axelera user guide you mention?

The guides are not public yet. Once we are done, we will share with the community. In the meantime, if you install using the instructions at Overview | DeGirum Docs, you should be able to run the examples that work on cloud locally also by changing @cloud to @local.

degirum sys-info

Hi ​@shashi.chilappagari , I ran into some issues. I have followed the installation instructions for linux, and also followed the driver installation instructions for Axelera here since degirum sys-info did not show Axelera device on the machine. I noticed that one package had to be downgraded (metis-dkms 1.0.2 - was using 1.0.3 before). Note that I had previously installed the latest Voyager SDK to my machine.

Anyway, after going through the process in the driver installation page without any errors and hard-power-cycling the board, I still don’t see the Metis device.

Moreover, now even the Axelera sdk has issues refreshing the device as below:

I have a feeling this may be due to the older version of runtime used by the Degirum (that we installed during driver installation?)

Hi ​@saadtiwana 

Thanks for sharing this. I will loop in our SW engineer who can help with this issue. One nice thing I see from your logs is that the Rockchip runtime RKNN is picked up by PySDK. This will allow you to use same codebase to run some models on the built-in NPU of the RK3588 (which has 6TOPS and can run detection models at >60FPS).

Btw, if i switch back to metis-dkms=1.0.3, I can use the hardware with voyager-sdk again, but still does not appear in degirum sys-info . For your info, my M.2 card is running the latest firmware, from the latet voyager-sdk release:

Yes, you are right, and I noticed that too earlier. In fact, i believe they are appearing because I had previously installed the Rockchip drivers too as per your driver installation guide :) It’s something I will try as well. That said, I do need to get the models running on the Axelera hardware anyway, so help in that regard will be appreciated. Ideally, if degirum works with the latest Axelera runtime, that will be ideal so I don’t need to switch between things continuously ;) Thanks.

Hi ​@saadtiwana 

We will definitely focus on getting it to work locally for you. In the initial stages, it is quite common to have versions not in sync, but we are working with Axelera to make sure PySDK releases stay in sync with Axelera updates. Good thing is that we have the same system as you, so we might bother you with same questions regarding the exact setup you have. Once we get it work internally with the version you have, we will make a new release. Our hope is that we get things to work soon so that you don’t have to switch. Meanwhile, please feel free to browse pysdkexamples as the code for cloud and local is same except for 1 line.

​@shashi.chilappagari Thanks, I was able to get the code working on my board with the RKNN backend and getting a very decent 25-30fps with the “yolov5s_relu6_coco--640x640_quant_rknn_rk3588_1” model. I did notice a fairly high CPU usage though  (~50% across cores) via htop, which maybe due to the pre/post processing? not sure.

Anyway, this is still quite amazing how easy it was to get it all running. Next I am looking forward to getting some models running using the Axelera backend locally. Pls do let me know when there’s an update on that. Thanks.

Hi ​@saadtiwana 

Glad to hear that you got PySDK working locally on RKNN. It should give you an idea of how to use the APIs. Yes, the high CPU usage is due to pre and post processing (mainly decoding and resizing). Anyways, there are a lot of optimizations that are relatively easy to do (like the ones using gstreamer that you have been trying). Once you get things working, we can guide you on optimizing the performance. 

Regarding Axelera: we replicated your issue and confirmed that it is due to PySDK supporting runtime 1.3.1 and your system having runtime 1.3.3. Internally, we were able to build a new PySDK that supports 1.3.3 and it passed all tests. An official release of PySDK with support for 1.3.3 will be made mid next week. I will keep you posted.

​@shashi.chilappagari I spent quite some time with the degirum sdk over the weekend, using the RKNN NPU. I am looking forward to your updated release this week to start experimenting with Axelera device using Degirum. Overall, so far I am happy with the ease of use that the Degirum sdk brings (and also the interoperability between different hardware, which is quite amazing). I have a few questions:

1. One question I have is, I notice that in the zoo you have a few of the Rockchip devices (3566, 3568, 3588) but the RKNN toolkit2 that I downloaded as the driver actually supports several other Rockchip devices as well which are not in your list. Let’s say if I were to use, for example, a RK3576 device (supported by RKNN toolkit2), can I still use Degirum with it? Would the models have to be compiled differently?
2. Secondly, I saw in another post that Degirum may move to a subscription model for the online compiler at a later stage. How about the pysdk itself? For non commercial/hobbyist/educational projects, will the users be able to continue using degirum with a subscription? Will they still need to purchase a subscription to compile models for use with degirum?
3. Can I use a model compiled in the Axelera sdk with Degirum directly? (Or compiled in RKNN toolkit2 for that matter, and so on) Or the compilation MUST happen via the degirum compiler?
4. For setting up a new device with degirum (inference), must I connect it to the internet for setting up or can I do that offline also? For devices without any form of internet connectivity, for example.

Btw, I applied for early access to the online compiler to try it out (same user name as here)

Ah, nice work ​@saadtiwana! Great to hear how it’s going using the AI Hub. Thanks for sharing!

Hi ​@saadtiwana 

Very happy to see that you are having a positive experience with PySDK, Our team is working hard to release new PySDK soon and I will keep you posted. Please see my answers below:
 

1. One question I have is, I notice that in the zoo you have a few of the Rockchip devices (3566, 3568, 3588) but the RKNN toolkit2 that I downloaded as the driver actually supports several other Rockchip devices as well which are not in your list. Let’s say if I were to use, for example, a RK3576 device (supported by RKNN toolkit2), can I still use Degirum with it? Would the models have to be compiled differently?

Our experience with RKNN shows that integrating a device that supports the same RKNN runtime is relatively straightforward. We showed 3566, 3568 and 3588 as proof of concept. If there is demand for RK3576, we can add it as well.

1. Secondly, I saw in another post that Degirum may move to a subscription model for the online compiler at a later stage. How about the pysdk itself? For non commercial/hobbyist/educational projects, will the users be able to continue using degirum with a subscription? Will they still need to purchase a subscription to compile models for use with degirum?

PySDK is and will remain free for non-commercial/hobbyist/educational projects. We introduced the enterprise PySDK license to provide dedicated support (as opposed to community support) for companies/use cases that need help developing and deploying edge AI applications as there was a lot of demand for dedicated support. Compiler may move to a subscription model because it uses quite a bit of compute resources in the cloud and due to the overwhelming response we saw in the past 6 weeks, there is no easy way to provide it for free to all users (you will see that our cloud compiler supports multiple HW options). So, to answer your question: yes, even non-commercial/hobbyist/educational projects need subscription for cloud compiler. We will of course consider feedback from enthusiastic community members like yourself to make sure the service is affordable to end users while making sure we can pay our bills 😀

1. Can I use a model compiled in the Axelera sdk with Degirum directly? (Or compiled in RKNN toolkit2 for that matter, and so on) Or the compilation MUST happen via the degirum compiler?

You can compile outside and use it with PySDK. However, PySDK needs a model JSON to be configured properly so that pre-processor and post-processor work properly. We have user guides and docs that explain how to do this but as always, we are here to help users with any problem they might face.

1. For setting up a new device with degirum (inference), must I connect it to the internet for setting up or can I do that offline also? For devices without any form of internet connectivity, for example.

PySDK can work without any internet connection. All online tools like cloud inference, cloud model zoos etc are tools we provide so that users can get started quickly without getting stuck in hardware setup issues and finding models. But once they have things working, compute and models can stay completely local. 

We received your application for cloud compiler access and will soon get back to you. Thank you for taking the time to work with PySDK and asking a lot of insightful questions that I am sure the entire community benefits from.

Hi ​@saadtiwana, we’ve just granted you access to the DeGirum Cloud Compiler. I’ve just emailed you some onboarding information. Please let me know if you have any questions or need any support! 😊

Hi ​@saadtiwana 

We just released our latest pySDK (version 0.18.0) with support for Axelera 1.3.3. Please give it a try and let us know if you encounter any issues.

Hi ​@shashi.chilappagari , ​@alexandra ,

Awesome, I will start testing the new version tonight. And thanks a lot for granting access to the online compiler, I will be trying that out in the coming days and let you know if I run into any issues. 

Btw, I noticed that the compiler allows up to 1280x1280 input image size on the models. I am curious to test and understand the pros and cons, performance differences between running a model at bigger input size versus tiling (which your sdk also supports). 

Looking forward to more testing! Thanks again!

Hi ​@saadtiwana 

regarding running model at bigger input size vs tiling: if the model is originally trained at higher resolution, it is most likely that it will be better. Tiling can have some performance degradation because it needs tiles to overlap. For some hardware, efficiency increases with input size (up to certain size), so model at higher resolution can be better from an FPS point of view as well. However, these are not universal, and we need to test for your use case. But we have all tools to make such analysis simple.

Hi ​@shashi.chilappagari I updated to the latest degirum version using pip’s upgrade method, and then also updated the Axelera driver (both following the instructions on the Degirum. However, somehow the Metis card is not getting detected by degirum sys-info

I did hard-power cycle (totally removing power) the board as mentioned in the degirum docs, even several times, but no luck. I also refreshed the Axelera device and could see it from the voyager-sdk

How do I go about troubleshooting this? To recap, I am using the Aetina RK3588 with Metis M.2.

Hi ​@saadtiwana 

I reported this internally and our engineer responsible for Axelera integration will help you.