How to make a vrm avatar make a real pose ? （It is not an issue but more like a suggestion）

[cyberpsyche]

It is not an issue but more like a suggestion. I could import a VRM model in a godot project based on the gorgeous vrm addon, but besides some pre-defined "blink" expressions I do not know how to make the avatar do an actual pose. Is there a code example or a manuscript to teach us how to really use the vrm model in godot ? Where can the godot-vrm users find it ?

Thanks very much!

[fire]

Since you asked, let us spend some time writing a summary of our current work on making the avatar do an actual pose.

Will post as soon as we can.

[cyberpsyche]

Very glad to hear that. Thank you !

[cyberpsyche]

@fire Hi fire, is there any progress on this topic ?? I hope this post is not annoying you or interrupting your ongoing project, but since several weeks past, I wonder when and how can we do something really interesting based on your vrm model control api. Thanks again!

[fire]

Over the last weeks we and the Godot Engine team merged several prs.

• gltf: Fix validation errors due to chunk padding and empty skins.
• Refactored Node3D rotation modes
• Implement animation compression
• Implement Animation Blend Shape Tracks
• Add scene Post-Import Plugin support.
• Implement import actions for pos/rot/scale animation tracks
• Remove REST transform influence in skeleton bones
• Remove animation 3D transform track, replace by loc/rot/scale tracks.
• Use an intermediate animation importer

We are working on VRM posing.

[fire]

https://github.com/vss-project/vss and https://github.com/you-win/openseeface-gd

Also we did some work with the vss and openseeface-gd projects.

[cyberpsyche]

Awesome !!!

I have already noticed the openseeface-gd project, but the vrm add-on in that project is not identical to godot-vrm project, and openseeface-gd is only control the head bone of a vrm model. I wonder is there a demo project to utilize all the potentials of godot-vrm with control all the bones of a vrm model. Also I wonder that could we control the fingers of a vrm model ?

[fire]

We updated Godot VRM to the new Godot Engine 4 that allows importing VRMs from the game instead of the editor.

Other things directly related to animation are in the development pipeline.

[fire]

The 4.0 code now uses gltf document extensions.

There are two in-progress approach to retargeting:

1. forward kinematics retargeting
2. inverse kinematics retarging

There is an improved statemachine pr. godotengine/godot#61196

[fire]

I have also made a proof of concept mediapipe to osc binary.

https://github.com/V-Sekai/mediapipe

Unsolved problem to convert mediapipe skeleton to vrm skeleton.

[fbcosentino]

Is there a way to apply animation to the skeleton externally?

I tried the following:

• In blender, add animations and export the same model as both .vrm and .glb
• Import VRM model via plugin -> works perfectly, spring bones also work, animations are not there
• Import glb, animations are there but obviously vrm features such s springbone are not there
• Copy AnimationPlayer node from the glb version and paste into the vrm version

animation names and bone names are correctly identified and the model does animate (with springbones working as well), but animations are completely broken, looks like transforms work in a different way?

[fbcosentino]

animation names and bone names are correctly identified and the model does animate (with springbones working as well), but animations are completely broken, looks like transforms work in a different way?

It seems it's related to rest pose. Testing with an animation where the first frame is already transformed (that is, not starting in rest position) importing from glb works fine as it knows the rest pose. Importing as vrm the first frame is taken as reference for rest and transforms are applied adding to that.

Tested with an animation where the first frame had all bone transforms reset, and transferring the AnimationPlayer glb->vrm worked fine

[fire]

We wrote a blog entry for this https://godotengine.org/article/animation-retargeting-in-godot-4-0

Let me know if this helps

[icynic]

I've noticed that the sample model of AliciaSolid has good physics in godot engine. But other models I downloaded from the Internet don't have physics at all.
Is it because the GDscript of the secondary node? If so, how did it do that?
Can I implement physics for other models in the same way, or maybe by using spring bones?

[fire]

This is a good question.

We use vrm spring bones.

They're also described here. godotengine/godot-proposals#5605

[fbcosentino]

If you have a model with no physics and you want to add the spring bones yourself, you can use blender with the VRM plugin: https://vrm-addon-for-blender.info/en/

It's a bit cumbersome so you're going to have to spend some time reading about how collision works with VRM spring bones, as you need a. add spring bone information to the bones you want physics (e.g. hair, clothes) and b. you need colliders so clothes and hair don't go into the body etc

The bright side is you can start with any model in any format (fbx, glTF, etc) and will be able to export VRM working in Godot - you don't need to start with a VRM model itself.

[icynic]

Thanks for your suggestions. And sorry for my negligence. My problem has been solved.

[fire]

Let me know if anything is pending to do here.

[cyberpsyche]

@fire Hi Fire, if I have a godot-vrm model in a Godot 4 scene, how to let the VRM model make a motion by gdscript? For example, if I want make the VRM model raise left arm or blink its right eye or open its mouth to say "AEIOU", how to do it？

I tried this code :

var vrm_model = $Godette_vrm_v4
var skeleton = vrm_model.get_node("GeneralSkeleton")
print(skeleton.get_bone_names())

But the editor prompt a error message for "nonexistent function get_bone_names()" .

How to pick up a specific bone via gdscript ?

Thanks.

[fire]

I don’t godot has ever had a get bone names. Godot Skeleton has a get bone count and you can use that index to get a bone name. use find bones to get an index from name? What tutorial are you using?

[cyberpsyche]

I can't find a up-to-date tutorial on the internet. So I asked ChatGPT :P

Is there an appropriate tutorial or a demo project you could suggest?

And if I want to apply a VMD file to the godot-vrm model, how can I do it? Where can I find a demo project for this ?

Thanks!

[fire]

We had a working demo of vmd to vrm playback in godot 3 but it broke https://github.com/V-Sekai/godot-miku-miku-dance-vrm-vsekai/tree/master.

No estimates on the feature completion timeline.

[lyuma]

@cyberpsyche ah, you're on Godot 4, right?
Well, you're in luck. Godot 4 includes Animation retargeting, and this VRM importer is compatible with Godot's humanoid format! So an imported VRM comes ready to accept any humanoid animation Godot can import.

So you could take any animation in fbx (requires a mesh) or gltf format, and on the left panel where it says Import As scene, select AnimationLibrary.
https://docs.godotengine.org/en/stable/tutorials/assets_pipeline/retargeting_3d_skeletons.html

For converting from VMD, if the importers in godot are not working, you could consider using a Blender conversion and export to FBX. The process will be the same as you might find in tutorials for Unity or VRChat, where the end result is ideally a gltf (or fbx)

[fire]

You'll need to change the structure of the PMX character with the hips joints to be compatible with VRM. I don't have the exact method on hand.

[cyberpsyche]

Thanks @lyuma and @fire, ultimately I could bind a .fbx motion file to an imported VRM model. It's great to see the VRM character to perform a motion (except the arms and legs will occasionally penetrate the clothes).

I wonder how to make a VRM model to do some 'innate' actions , for example, to blink eyes or open mouth for emulation to say "AEIOU" words while do the .fbx motion animation. After all, let the VRM model do a long time dance motion without even blinking his/her eyes once seems very weird. How can I do this just via GDScript, not by using an animation file in Godot 4?

[lyuma]

I have since implemented blend shape and bone animations for eyes and mouth, into the AnimationPlayer. So to implement this, it will be necessary to add a few BlendSpace2D (for the eye animations) or BlendTree + Add3 node (for the mouth / facial expressions).

Or, perform the animation by hand in code by reading the tracks from the AnimationPlayer to see which keyframes are being set.

Anyway, since Godot's animation system has greatly improved in 4.x, I think it is finally time to close this issue. I would encourage people using Godot 3 to consider moving to Godot 4.