annotations for custom training ?

[DHG-Dav]

Hello again !
I've been busy since my last message, on other projects (i'm still new in this field and python programming so everything takes forever), i'm now back to trying to make my ACGAN to generate manga pictures. (i'm pretty much a newbie in this field, i hope i won't ask stupid questions ! )
Your pre-trained model works wonderfully in most cases, but i'm getting a bit lost in your code, is there a fonction to record .json with keypoints after a detection somewhere ?
Also, your model is trained on 25 keypoints right ? so there is no easy way (like a script already in the project ?) to actually generate results that could be manually corrected if a few keypoints are wrong and then used to re-train the model ? (your pre-trained model struggle on partial characters, like bust shots, and with foreshortening/strong perspectives, i guess it's not because of the architecture but just because of the dataset, so i'd like to train it a bit more with an extended dataset)
Thank you !

[ShuhongChen]

fonction to record .json with keypoints after a detection

Not a direct json, but the info you need is in the output printed from this, which shows (x,y) coords of each 17 named kpts; if you were to hack your own json writer, I'd start here

25 keypoints ... no easy way to actually generate results

Yes, the model outputs 25 kpts (17 coco + 8 midpoints); if you're interested in the coco ones, they're just the first 17 of the 25

your pre-trained model struggle on partial characters, like bust shots, and with foreshortening/strong perspectives, i guess it's not because of the architecture but just because of the dataset

Yeah we don't handle partial characters; fullbody foreshortening can be fixed by more data, but partial can't be fixed without changing the architecture. IIRC, partial is handled by also outputting a visibility score, but we don't do this since we assume fullbody

train it a bit more with an extended dataset

If you want to train a model that can handle occlusion/partial, you can also augment the current dataset by cropping; again, this still needs architecture change though. Overall if you're interested in retraining something, I'd honestly look at a more standard repo like mmcv (I don't really like detectron2); our training code is admittedly a mess...

That's not to say our repo is 100% useless. What you're saying about using our detector to quickly annotate more data is a good idea; also, if you can hack our pretrained classifier into like an mmcv backbone, I'm guessing that'll make significant difference.

Thanks for your interest in our work, hope this clarifies things!

[DHG-Dav]

Thank you very much, once more for the clear explanation, suggestions and help ! ^^
My bad for the keypoints !
I'll look into MMCV, i already tried imageAI, openpose and detectron by the past but the results weren't up to my expectations, i tested openpose with your dataset (just in case) but it underperforms compared to yours. I still have to test models like bbpose and omnipose (or unipose ), i'll keep you updated if you want.
Your project is awesome, your code and explanations are more clear than a LOT of projects i've seen (and i get to learn pytorch ^^ )
I edited the script to generate a text file with the same name than the image file, following the layout of your JSON, so anyone could use it to generate these files and then just concatenate them all into a json effortlessly. My code is gross, i am aware of it, i'm still learning, and it's tested only on windows, but i join it to this message, just in case. (i change .py to .txt because of github limitations.) (oh, and it works on directories now instead of single image)
pose_estimator.txt
character_segmenter.txt

Thank you !