Purethermal 2

[nsknyc]

I noticed that there's a config for 'groupgets' that sets the proper resolution for a purethermal 2 and lepton 3.5 module.
Surprisingly when i hit the stream video it activates the camera (light blinks quickly), but there is no image (it just stays white). This is the farthest I've gotten with this camera on android so far. I've tried UVCcamera and its variants without any success. Unfortunately, the only android device I have is a DJI smart controller which, If I'm not mistaken, runs on Android 7.1.2. I have never developed for android before, so please bear with me.

First, I'd like to point out that I get a build failed under android studio. I get a
Process 'command '/Users/nsk/Library/Android/sdk/ndk/21.1.6352462/ndk-build'' finished with non-zero exit value 2
I got figured out that it has to do to app/src/jni/Android.mk includes that don't exist. After commenting those out, one needs to allow_missing_deps, after this build succeeds.
I'm aiming for this path, because I figured I could get more info debugging this way.

I know that there are a bunch of factors here, dji device(probably a custom android version), thermal camera(luckily its very well documented), and android version on host device, but any help on directing me how to debug this would be greatly appreciated.

[Peter-St]

Hello nsknyc,

first lets find out, if the camera is an uvc device:
On a Linux System run this command:

lsusb -v -d xxxx:xxxx
The first 4 x are the product and the second 4 x are the vendor id.
If you can run this command successfully, you can attach the output.

So far,

Peter

[nsknyc]

Ok, some corrections. I don't want anyone to be misled. DJI smart controller is essentially an Android tablet, but it doesn't come with google play preinstalled. So to get this app, I needed to use an 'apk downloader', and due to that I was only able to get an older version of the application. So I decided to install (jumping a couple of hoops) google play and get the updated app through there.
I was able to successfull connect the camera with the new version of the apk. The app now streams the thermal image, albeit not optimally, but I would love to fix that!
As for the build, allow_missing_deps isn't the solution, but commenting out
#include $(PROJ_PATH)/libusb/android/jni/Android.mk
#include $(PROJ_PATH)/libUsb_Support/Android.mk
#include $(PROJ_PATH)/android-libjpeg-turbo/Android.mk
#include $(PROJ_PATH)/jpeg8d/Android.mk
#include $(PROJ_PATH)/libyuv/Android.mk

But by doing so the app will crash if you try to enter "set up the camera device"
So far this is where I left off. Again, sorry that I may seem 'all over the place' jumping in to a new enviroment and project triggers my ADHD. If libusb is needed for this build, then I'll try to use the modified version used in the getthermal project. https://github.com/groupgets/GetThermal
"GetThermal uses a modified version of libuvc for camera image download and control. Even if you have libuvc on your system already, you will have to build the fork."
Btw, this is the official application one can use with the purethermal 2 module.

[nsknyc]

Here is my lsusb off of virtualbox. The osx version apparently doesn't understand what Verbose means.
lsusb.txt

[Peter-St]

Hello nsknyc,

from your outreaded camera specifications I can see, that the termal camera uses different image formats than standard uvc cameras:
The output camera format is described with the guid number:

guidExtensionCode {2d317470-656c-2d70-6f65-6d2d30303030}

I don't know, how to solve this up to now, because this guid is not described anywhere.
Perhaps some information can be found in the existing source code:
https://github.com/groupgets/GetThermal
I hope I can spend some time on it on the comming weekend.

So far,

Peter

[Peter-St]

Ok, some corrections. I don't want anyone to be misled. DJI smart controller is essentially an Android tablet, but it doesn't come with google play preinstalled. So to get this app, I needed to use an 'apk downloader', and due to that I was only able to get an older version of the application. So I decided to install (jumping a couple of hoops) google play and get the updated app through there.
I was able to successfull connect the camera with the new version of the apk. The app now streams the thermal image, albeit not optimally, but I would love to fix that!

Here I need some informations:

• Which setting have you used to perform the stream? (AltSetting, MaxPacketSize, Videoformat, Camera FormatIndex, Camera FrameIndex,...)

The termal camera uses some image formats, which my app actually does not support: UYVY (I think I can add this format). That's the reason, why the stream is actually not optimal.

As for the build, allow_missing_deps isn't the solution, but commenting out
#include $(PROJ_PATH)/libusb/android/jni/Android.mk
#include $(PROJ_PATH)/libUsb_Support/Android.mk
#include $(PROJ_PATH)/android-libjpeg-turbo/Android.mk
#include $(PROJ_PATH)/jpeg8d/Android.mk
#include $(PROJ_PATH)/libyuv/Android.mk

But by doing so the app will crash if you try to enter "set up the camera device"

The app crashes, because it can not find the native librarys to load.
You can try to comment out the whole native build part in the apps build.gradle file: "comment out the whole externalNativeBuild, task ndkBuild, task ndkClean". (You do not need to run the ndk part, because the native libs are already precompiled in the libs folder).
Before you run the app, be sure, that all native *.so files are present in the libs folder (if not, copy the precompiled ones from this repo to it).

So far this is where I left off. Again, sorry that I may seem 'all over the place' jumping in to a new enviroment and project triggers my ADHD. If libusb is needed for this build, then I'll try to use the modified version used in the getthermal project. https://github.com/groupgets/GetThermal
"GetThermal uses a modified version of libuvc for camera image download and control. Even if you have libuvc on your system already, you will have to build the fork."
Btw, this is the official application one can use with the purethermal 2 module.

[nsknyc]

Thank you for the update. I've been a bit busy work, but I'll be on top of this again tomorrow.

[nsknyc]

Hmm, these libs were not in my folders, I'm pretty sure I cloned the repo properly. I must have done a clean and it emptied out.

Before you run the app, be sure, that all native *.so files are present in the libs folder (if not, copy the precompiled ones from this repo to it).

I have a compilation issue, missing symbol humer.UvcCamera.R
**** strike this compilation issue****
I had change the name of the app and missed a couple of things. I wanted to run two version and test things.

As for the settings I used they are
ALT_SETTING:2
MaxPacketSize:962
FormatIndex:1
Videoformat:YUY2
FrameIndex:1
ImageWidth:160
ImaageHeight:120
FrameInterval:1111111
PacketsPerRequest:1
ACTIVE_URBS:1
LibUsb Support: false

After experimenting for a while the Frame index changes the type of video output. I honestly thought that was something done by the getthermal app.

[Peter-St]

Hmm, these libs were not in my folders, I'm pretty sure I cloned the repo properly. I must have done a clean and it emptied out.

Yes, till the build, the native libs folder is cleaned and rebuild again. Try to exclude the whole native parts in the apps build.gradle file. -> download the repo as a zip file and copy the native libs manually to its directory.

Before you run the app, be sure, that all native *.so files are present in the libs folder (if not, copy the precompiled ones from this repo to it).

I have a compilation issue, missing symbol humer.UvcCamera.R
**** strike this compilation issue****
I had change the name of the app and missed a couple of things. I wanted to run two version and test things.

As for the settings I used they are
ALT_SETTING:2
MaxPacketSize:962
FormatIndex:1
Videoformat:YUY2
FrameIndex:1
ImageWidth:160
ImaageHeight:120
FrameInterval:1111111
PacketsPerRequest:1
ACTIVE_URBS:1
LibUsb Support: false

Ok, the maxPacketSize and Altsetting looks to be allreigh. The FormatIndex is also right, but the Videoformat is not the right one: It should be UYVY but not supported from my app up to now.

The FrameIndex, ImageWidth, ImageHeight and FrameInterval are allreigh.
The Values for PacketsPerRequest and ActiveUrbs can be raised at least to 2 for both. You can try to raise them to perhaps 16 --> this can improve the quality of the stream.

You have set the LibUsb Support to false. If you get a fluid stream, you can stay at false (The stream is performed only over the java side, which is a bit slower than using the libusb library, which runs in pure C code and is faster due to this behavior).

After experimenting for a while the Frame index changes the type of video output. I honestly thought that was something done by the getthermal app.

[nsknyc]

The FrameIndex, ImageWidth, ImageHeight and FrameInterval are allreigh.
The Values for PacketsPerRequest and ActiveUrbs can be raised at least to 2 for both. You can try to raise them to perhaps 16 --> this can improve the quality of the stream.

I have increased those before. The stream does noticeably improve, but the issue here is definitely the colors and resolution.

I've spent all day looking over the code, and realized that you did a java implementation of libuvc (essentially). Good stuff! I've also have been using purethermal2 board's firmware source code as a reference. I'm not familiar with uvc, but I'm trying my best to catch up.

I confirmed some of the info off of this file. I'm not sure why is says 80,60 max frame size, but the lepton module is definitely 160,120. The getthermal app definitely pulls a higher resolution. Perhaps it communicates with the purethermal board to increase the resolution.
https://github.com/groupgets/purethermal1-firmware/blob/master/Src/usbd_uvc_if.c

-- edit --
I wanted to add that auto detect still makes the app crash. I suspect that it tries to do a libusb check?

[Peter-St]

Hello nsknyc,

can you plz try out the following apk with UYVY support:

https://drive.google.com/file/d/1c0EJ8os-U24djrTg9K_GEDYOgBldZhUD/view?usp=sharing

Select under VideoFormat UYVY
The rest of the other values you can keep.

So far,

Peter

[nsknyc]

No stream, white background. The camera is activated though. When it starts streaming it goes from 1 second blinking to a very fast blinking pattern.

[Peter-St]

Ok,

let's take a look at the frames:
With your above settings....
FormatIndex:1
Videoformat:YUY2 / UYVY
FrameIndex:1
ImageWidth:160
ImaageHeight:120
... each frame should have a size of 38400 bytes.

Under the Button "Set Up The Camera Device" you can also analyse your camera stream. First post an output of the control transfer to your camera.

Next perform a testrun and post the output of 1 frame received and frames for 5 seconds. You can try out different values for PacketsPerRequest and ActiveUrbs. When every frame has a length of 38400 bytes, your stream is allright.
Also try it out with and without LibUsb Support.

The termal camera has a lot more functions under the Control_Interface than a regular usb camera.
You can also try to test the other format index's to receive the frames (these format are not supported in the picture mode, but you should get an output under the Testrun option.

Eventually attach some screenshots with the outputs you receive, or write it down to a txt file.

So far,

Peter

[nsknyc]

Would you like all the data from the test (1 frame(YUY2 & UYVY) and 5 frames (YU2 & UYVY)) including the hex strings on the bottom?
Setting libusb will crash the application. Same as doing automatic detection.

[Peter-St]

Basically only the frame lenght.
The hex values in their strings should look different to each other, then the camera stream works --> 0x81 0x81 0x81 0x81..... in the hex string is an invalid frame.

Next point would be to save a frame with a lenght of 38400 to a file and display it on a desktop pc, to see, if it's a valid frame.

When the LibUsb part is crashing, it should be debugged under Android Studio. The Automatic Detection so far is a beta up to now, so here a crash doesn't matter.
Where you still able to build the project without the native part? If yes, eventually post the crash log.

Otherwise to the missing repos included in the Android.mk file:
The were not present, because I only cloned them to the project and not included the downloaded zip file instead.

I have attached all native folders.

You should then be able to build the app under Android Studio. (I'm compiling this project under Linux as operating system, maybe win is throwing some other errors)

So far,

Peter

[nsknyc]

Basically only the frame lenght.
The hex values in their strings should look different to each other, then the camera stream works --> 0x81 0x81 0x81 0x81..... in the hex string is an invalid frame.

The hex streams do seem random. I'll be posting them soon.

Next point would be to save a frame with a lenght of 38400 to a file and display it on a desktop pc, to see, if it's a valid frame.

The frame lengths definitely hit 38400. I'll try to save these hex values somehow. I've only saved these through screenshot.

When the LibUsb part is crashing, it should be debugged under Android Studio. The Automatic Detection so far is a beta up to now, so here a crash doesn't matter.
Where you still able to build the project without the native part? If yes, eventually post the crash log.

Yes, libusb part is crashing. I wish I could debug through android studio unfortunately there's an issue on OSX that devices don't appear in the "device file explorer" same happens with android file transfer. That is something apart though.
I was able to build the project without the native part.

The were not present, because I only cloned them to the project and not included the downloaded zip file instead.
You have to download or clone the following repos for the native part

I included the libs from the zip file, but I'll include the sources and build them. Btw I'm on mac OSX. Let's see how it goes today.

[Peter-St]

The native libs are available now. You can try to compile them.

[nsknyc]

Build with native libs successful. Device explorer working now.
Crash when libusb is used and you hit "start the camera" under transmission start.

Caused by: java.lang.NullPointerException: Attempt to read from null array
at humer.UvcCamera.StartIsoStreamActivity.isoStream(StartIsoStreamActivity.java:1452)

I have an update. The camera does in fact support UYVY, but the output we get through the 'getthermal' app is raw data.

The GetThermal program will also acquire raw 14-bit data from Radiometric Leptons by default.

https://github.com/groupgets/purethermal1-firmware/wiki/Transferring-raw-(14-bit)-Lepton-data

The uvc frame format should be Y16.

[nsknyc]

@38293312 hmm, that doesn't seem like an app issue. It doesn't even detect the purethermal board. I suggest trying different cables make sure its USB OTG. Once you connect pt2 board android should detect it before even launching the app.

[nsknyc]

Ok so from StartIsoStreamActivity.java

2021-03-30 00:53:28.068 2077-2153/humer.uvc_camera I/UVC_Camera_Iso_Stream: calling convertUYVYtoJPEG java
2021-03-30 00:53:28.069 2077-2153/humer.uvc_camera D/From Native: tjCompress2 failed !!!!!!, ret = 0
2021-03-30 00:53:28.069 2077-2153/humer.uvc_camera I/UVC_Camera_Iso_Stream: jpgLength = -1
2021-03-30 00:53:28.069 2077-2153/humer.uvc_camera I/UVC_Camera_Iso_Stream: no Data received == NULL
2021-03-30 00:53:28.069 2077-2153/humer.uvc_camera I/UVC_Camera_Iso_Stream: jpegByteArray == null

[nsknyc]

@38293312 May you upgrade the purethermal firmware to version 1.3.0 and try again?
https://github.com/groupgets/purethermal1-firmware/releases

[Peter-St]

@38293312 please post in a seperate isssue, because otherwise it gets wired.
Pure Termal Issue #20
#20

Thx,

Peter

[Peter-St]

Ok so from StartIsoStreamActivity.java

2021-03-30 00:53:28.068 2077-2153/humer.uvc_camera I/UVC_Camera_Iso_Stream: calling convertUYVYtoJPEG java
2021-03-30 00:53:28.069 2077-2153/humer.uvc_camera D/From Native: tjCompress2 failed !!!!!!, ret = 0
2021-03-30 00:53:28.069 2077-2153/humer.uvc_camera I/UVC_Camera_Iso_Stream: jpgLength = -1
2021-03-30 00:53:28.069 2077-2153/humer.uvc_camera I/UVC_Camera_Iso_Stream: no Data received == NULL
2021-03-30 00:53:28.069 2077-2153/humer.uvc_camera I/UVC_Camera_Iso_Stream: jpegByteArray == null

Ok, turbo-jpeg failed.
I can handle this, but soonest on the weekend I think.

--> great you can build the repo now.

[Peter-St]

The uvc frame format should be Y16.

Ok, eventually I will add this format too.

[Peter-St]

Ok so from StartIsoStreamActivity.java

2021-03-30 00:53:28.068 2077-2153/humer.uvc_camera I/UVC_Camera_Iso_Stream: calling convertUYVYtoJPEG java
2021-03-30 00:53:28.069 2077-2153/humer.uvc_camera D/From Native: tjCompress2 failed !!!!!!, ret = 0
2021-03-30 00:53:28.069 2077-2153/humer.uvc_camera I/UVC_Camera_Iso_Stream: jpgLength = -1
2021-03-30 00:53:28.069 2077-2153/humer.uvc_camera I/UVC_Camera_Iso_Stream: no Data received == NULL
2021-03-30 00:53:28.069 2077-2153/humer.uvc_camera I/UVC_Camera_Iso_Stream: jpegByteArray == null

Problem should be solved. Rebuild the app to test the uyvy stream.

[nsknyc]

Built successful. Image looks fantastic. I have forked this project, because I have some ideas for some features. Of course, I feel that this project has a 'debug' type nature, which is what I love about it. I have just recently finished adding a feature (uart debug messages) to the firmware, so now I'll have time to focus on this project.

[nsknyc]

Ok. I think I'll close this, as the update has worked successful. @Peter-St thank you very much for your help. Hopefully this will help other PT users

[kubabuda]

@nsknyc what was final working configuration, that made PT display feed on Android-UVC correctly?
And should this work with current app version from play store?

[nsknyc]

I've been a bit side tracked with other projects, but last version that worked for me I compiled from source (after Peter updated the code). I didn't use the playstore version. I'm not sure if its updated. I have a new android device now, and plan to test it there. Unfortunately, I can't give an exact time frame atm.