jgltf model builder question #102
Comments
|
I'm aware of the responsibility that comes with publishing a library that people rely on. I hate it when things change randomly, in an incompatible way. I try to avoid that. There are some things that should be improved and extended (and made "better" and "more stable") in the project as a whole. (There are some pending PRs/changes (not API related, but bugfixes) that are not yet part of the last release). And there are things that should be better supported (i.e. made more convenient) in the However, a recent issue with an example for creating skinning data is, for example, in #94 (comment) I'm also aware that there should be more documentation that is based on Markdown/Diagrams/Examples (and not only the - sometimes admittedly pretty shallow - JavaDoc documentation). But I'll also have to allocate more time for that. |
|
Thank you for the very quick reply, i will take a look at the issue you linked. |
|
There are some low-hanging fruits for making things a bit "nicer". For example, the DefaultAccessorModel joints = AccessorModels.create(
GltfConstants.GL_UNSIGNED_SHORT, "VEC4", false,
Buffers.createByteBufferFrom(ShortBuffer.wrap(new short[]
{
0, 0, 0, 0,
0, 0, 0, 0,
0, 1, 0, 0,
0, 1, 0, 0,
0, 1, 0, 0,
0, 1, 0, 0,
0, 1, 0, 0,
0, 1, 0, 0,
0, 1, 0, 0,
0, 1, 0, 0
}
)));one could make a case for offering something that avoids this (unintuitive) |
|
Yeah that would improve it a bit but honestly it's already really good the way it is currently, I've managed to almost reproduce what i had a few years ago, in just a few hours so creating gltf models with this model-builder api is already great. However i ran into an issue with skins, the armature it exports looks exactly as I'd expect but the mesh does not display properly (It's just a single triangle in the middle of the model) without the armature, it looks correct. Here is my current code: import de.javagl.jgltf.model.GltfConstants;
import de.javagl.jgltf.model.creation.AccessorModels;
import de.javagl.jgltf.model.creation.GltfModelBuilder;
import de.javagl.jgltf.model.impl.*;
import de.javagl.jgltf.model.io.Buffers;
import de.javagl.jgltf.model.io.GltfModelWriter;
import de.javagl.jgltf.model.v2.MaterialModelV2;
import editor.definition.SeqDefs;
import editor.definition.SeqFrameDefs;
import editor.render.animation.experimental.SkeletonHierarchyGenerator;
import editor.render.animation.experimental.model.BoneModel;
import editor.rs.RSModel;
import lombok.Getter;
import lombok.Setter;
import lombok.SneakyThrows;
import org.joml.Matrix4f;
import org.joml.Vector3f;
import org.joml.Vector3i;
import java.io.IOException;
import java.nio.FloatBuffer;
import java.nio.IntBuffer;
import java.nio.file.Path;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@Getter
@Setter
public class GltfExporter {
private RSModel model;
private final SeqDefs seqDefs;
private final SeqFrameDefs seqFrameDefs;
// in my case a bone just has a local position, a parent (null for root bone) and a list of children
private List<BoneModel> bones;
// temporary, figure out a better way to handle this rather than storing it as a global variable
private DefaultNodeModel rootJointNode;
public GltfExporter(SeqDefs seqDefs, SeqFrameDefs seqFrameDefs) {
this.seqDefs = seqDefs;
this.seqFrameDefs = seqFrameDefs;
}
@SneakyThrows(IOException.class)
public void export(Path path) {
SkeletonHierarchyGenerator skeletonHierarchyGenerator = new SkeletonHierarchyGenerator(seqDefs, seqFrameDefs);
skeletonHierarchyGenerator.setModel(model);
bones = skeletonHierarchyGenerator.infer(824);
// this just inverts the x/z axis of the bone position and also scales it by 0.02 (same thing is done below for vertices)
bones.forEach(BoneModel::postProcess);
// this computes the global transform as: globalTransform = new Matrix4f().translate(globalPos);
// and the inverse bind transform as: inverseBindTransform = globalTransform.invert(new Matrix4f());
bones.forEach(BoneModel::computeMatrices);
DefaultMeshModel mesh = new DefaultMeshModel();
DefaultMeshPrimitiveModel meshPrimitive = getMeshPrimitiveModel();
mesh.addMeshPrimitiveModel(meshPrimitive);
DefaultNodeModel nodeWithMesh = new DefaultNodeModel();
nodeWithMesh.addMeshModel(mesh);
DefaultSkinModel skin = getSkinModel();
nodeWithMesh.setSkinModel(skin);
DefaultSceneModel scene = new DefaultSceneModel();
scene.addNode(nodeWithMesh);
scene.addNode(rootJointNode);
GltfModelBuilder gltfModelBuilder = GltfModelBuilder.create();
gltfModelBuilder.addSceneModel(scene);
DefaultGltfModel gltfModel = gltfModelBuilder.build();
GltfModelWriter writer = new GltfModelWriter();
writer.writeEmbedded(gltfModel, path.toFile());
}
private DefaultMeshPrimitiveModel getMeshPrimitiveModel() {
DefaultMeshPrimitiveModel meshPrimitive = new DefaultMeshPrimitiveModel(4);
DefaultAccessorModel vertices = getVerticesAccessorModel();
DefaultAccessorModel indices = getIndicesAccessorModel();
DefaultAccessorModel weights = getWeightsAccessorModel();
DefaultAccessorModel joints = getJointsAccessorModel();
meshPrimitive.setIndices(indices);
meshPrimitive.putAttribute("POSITION", vertices);
meshPrimitive.putAttribute("JOINTS_0", joints);
meshPrimitive.putAttribute("WEIGHTS_0", weights);
// just use a default material for all triangles for now
MaterialModelV2 materialModel = new MaterialModelV2();
materialModel.setBaseColorFactor(new float[]{0.15f, 0.15f, 0.15f, 1f});
meshPrimitive.setMaterialModel(materialModel);
return meshPrimitive;
}
private DefaultSkinModel getSkinModel() {
DefaultSkinModel skin = new DefaultSkinModel();
DefaultAccessorModel inverseBindMatrices = getInverseBindMatricesAccessorModel();
skin.setInverseBindMatrices(inverseBindMatrices);
Map<Integer, DefaultNodeModel> idToJointNode = new HashMap<>();
for (BoneModel bone : bones) {
DefaultNodeModel jointNode = new DefaultNodeModel();
// local position is defined as globalPosition - parentPosition (if parent is null then parentPosition = zero vector)
jointNode.setTranslation(bone.getLocalPositionAsFloatArray());
jointNode.setRotation(new float[]{0f, 0f, 0f, 1f});
idToJointNode.put(bone.getId(), jointNode);
skin.addJoint(jointNode);
}
// set children
for (BoneModel bone : bones) {
DefaultNodeModel jointNode = idToJointNode.get(bone.getId());
for (BoneModel child : bone.getChildren()) {
DefaultNodeModel childNode = idToJointNode.get(child.getId());
jointNode.addChild(childNode);
}
if (bone.isRoot()) {
rootJointNode = jointNode;
}
}
return skin;
}
private DefaultAccessorModel getVerticesAccessorModel() {
FloatBuffer verticesBuffer = FloatBuffer.allocate(model.vertexCount * 3);
for (int i = 0; i < model.vertexCount; i++) {
Vector3f vertex = new Vector3f(model.verticesX[i], model.verticesY[i] * -1, model.verticesZ[i] * -1).mul(0.02f);
verticesBuffer.put(vertex.x).put(vertex.y).put(vertex.z);
}
verticesBuffer.flip();
return AccessorModels.createFloat3D(verticesBuffer);
}
private DefaultAccessorModel getIndicesAccessorModel() {
IntBuffer indicesBuffer = IntBuffer.allocate(model.triangleCount * 3);
for (int i = 0; i < model.triangleCount; i++) {
Vector3i index = new Vector3i(model.faceIndicesA[i], model.faceIndicesB[i], model.faceIndicesC[i]);
indicesBuffer.put(index.x).put(index.y).put(index.z);
}
indicesBuffer.flip();
return AccessorModels.createUnsignedShortScalar(indicesBuffer);
}
private DefaultAccessorModel getWeightsAccessorModel() {
FloatBuffer weightsBuffer = FloatBuffer.allocate(model.vertexCount * 4);
for (int i = 0; i < model.vertexCount; i++) {
// each vertex will only be influenced by one bone in my case
weightsBuffer.put(1f).put(0f).put(0f).put(0f);
}
weightsBuffer.flip();
return AccessorModels.createFloat4D(weightsBuffer);
}
private DefaultAccessorModel getJointsAccessorModel() {
IntBuffer jointsBuffer = IntBuffer.allocate(model.vertexCount * 4);
for (int i = 0; i < model.vertexCount; i++) {
int jointIndex = findJointIndex(i);
jointsBuffer.put(jointIndex).put(0).put(0).put(0);
}
jointsBuffer.flip();
return AccessorModels.create(GltfConstants.GL_UNSIGNED_INT, "VEC4", false, Buffers.createByteBufferFrom(jointsBuffer));
}
private DefaultAccessorModel getInverseBindMatricesAccessorModel() {
FloatBuffer inverseBindMatricesBuffer = FloatBuffer.allocate(bones.size() * 16);
int idx = 0;
for (BoneModel bone : bones) {
putMatrixToBuffer(bone.getInverseBindTransform(), idx * 16, inverseBindMatricesBuffer);
idx++;
}
inverseBindMatricesBuffer.flip();
return AccessorModels.create(GltfConstants.GL_FLOAT, "MAT4", false, Buffers.createByteBufferFrom(inverseBindMatricesBuffer));
}
private void putMatrixToBuffer(Matrix4f m, int offset, FloatBuffer dest) {
dest.put(offset, m.m00());
dest.put(offset + 1, m.m01());
dest.put(offset + 2, m.m02());
dest.put(offset + 3, m.m03());
dest.put(offset + 4, m.m10());
dest.put(offset + 5, m.m11());
dest.put(offset + 6, m.m12());
dest.put(offset + 7, m.m13());
dest.put(offset + 8, m.m20());
dest.put(offset + 9, m.m21());
dest.put(offset + 10, m.m22());
dest.put(offset + 11, m.m23());
dest.put(offset + 12, m.m30());
dest.put(offset + 13, m.m31());
dest.put(offset + 14, m.m32());
dest.put(offset + 15, m.m33());
}
// could this be the problem?
// the joints array is: "joints" : [ 1, 3, 2, 8, 9, 10, 11, 12, 13, 4, 5, 6, 7 ]
// however this is not the order the bones (read joints) appear in, they're all sequential starting from 0 (root bone is first)
// So it'd be: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
// I guess the reason why they appear in this order has to do with their children
// As if i don't add any children bones then the order is sequential (goes from 1 to 13 which makes sense)
// if this is indeed the issue, then i guess i have to figure out the indices of the joints as they appear in the glTF file, somehow
private int findJointIndex(int vertex) {
for (int i = 0; i < bones.size(); i++) {
BoneModel bone = bones.get(i);
if (bone.influences(model, vertex)) {
return i;
}
}
throw new IllegalStateException(STR."Didn't find a bone that influenced vertex \{vertex}");
}
}glTF file that the code produces for my specific input (a mesh of a male character): glTF file that the same code produces if i comment out the following: //DefaultSkinModel skin = getSkinModel();
//nodeWithMesh.setSkinModel(skin);
//scene.addNode(rootJointNode); |
|
It's a bit difficult to analyze the code, without the necessary classes ( Also, without the original model, there will always be some guesses involved. Things that I can say for sure:
When using identity matrices for the IBM, and fixing the JOINTS_0 component type, the result is the following: Yeah, it looks wrong, but with the right IBMs, it might not be totally wrong... I guess. An aside: I actually took some time recently, and did a few experiments for extending the builder functionality. The current state is in master...builder-extensions , for example, for adding morph target support to the builder classes ( 4bc4c46 ). But all this is still in flight. Better builder support for skinning and animations and all that may involve quite a bit of work and thought, and may take some time. But maybe you have some ideas or feedback. The inverse bind matrices can be printed with something like this: package de.javagl.jgltf.test.issue102;
import java.io.IOException;
import java.nio.file.Paths;
import java.util.List;
import java.util.Locale;
import de.javagl.jgltf.model.AccessorData;
import de.javagl.jgltf.model.AccessorFloatData;
import de.javagl.jgltf.model.AccessorModel;
import de.javagl.jgltf.model.GltfModel;
import de.javagl.jgltf.model.SkinModel;
import de.javagl.jgltf.model.impl.DefaultSkinModel;
import de.javagl.jgltf.model.io.GltfModelReader;
public class ReadIssue102
{
public static void main(String[] args) throws IOException
{
GltfModelReader r = new GltfModelReader();
GltfModel model = r.read(Paths.get("./data/issue102/test/test.gltf"));
List<SkinModel> skinModels = model.getSkinModels();
DefaultSkinModel skinModel = (DefaultSkinModel) skinModels.get(0);
AccessorModel am = skinModel.getInverseBindMatrices();
AccessorData ad = am.getAccessorData();
AccessorFloatData afd = (AccessorFloatData) ad;
System.out.println(afd.createString(Locale.ENGLISH, "%8.3f", 1));
}
}... which will print all zeros for these matrices. |
|
Thank you again for your quick reply
Here is the import editor.cache.model.sequence.TransformType;
import editor.rs.RSModel;
import javafx.geometry.Point3D;
import lombok.EqualsAndHashCode;
import lombok.Getter;
import lombok.Setter;
import org.joml.Matrix4f;
import org.joml.Vector3f;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
@Getter
@Setter
@EqualsAndHashCode(onlyExplicitlyIncluded = true)
public class BoneModel {
private int id;
private boolean root;
private BoneModel parent;
private List<BoneModel> children = new ArrayList<>();
private Vector3f globalPos;
private Vector3f localPos;
@EqualsAndHashCode.Include
private TransformType type;
@EqualsAndHashCode.Include
private Set<Integer> labels = new HashSet<>();
private Matrix4f localTransform;
private Matrix4f globalTransform;
private Matrix4f inverseBindTransform;
// could probably name this better
public void removeLabelsFromParents() {
removeLabelsFromParent(this.parent);
}
private void removeLabelsFromParent(BoneModel parent) {
if (parent == null) {
return;
}
parent.getLabels().removeAll(labels);
removeLabelsFromParent(parent.parent);
}
public boolean influences(RSModel model, int vertex) {
for (int label : labels) {
int[] vertexGroup = model.getVertexGroup(label);
if (vertexGroup == null) {
continue;
}
for (int v : vertexGroup) {
if (v == vertex) {
return true;
}
}
}
return false;
}
public void postProcess() {
globalPos = new Vector3f(globalPos.x(), globalPos.y() * -1, globalPos.z() * -1).mul(0.02f);
}
public void computeMatrices() {
Vector3f parentPos = parent != null ? new Vector3f(parent.globalPos) : new Vector3f();
// doesn't need to really be stored, can extract this from localTransform but keep it for now
localPos = new Vector3f(globalPos).sub(parentPos);
localTransform = new Matrix4f().translate(localPos);
globalTransform = new Matrix4f().translate(globalPos);
inverseBindTransform = globalTransform.invert(new Matrix4f());
}
// some utility methods
public float[] getLocalPositionAsFloatArray() {
return new float[] {localPos.x, localPos.y, localPos.z};
}
// delete later, only used for some debug code atm
public Point3D getOriginAsPoint3D() {
return new Point3D(globalPos.x, globalPos.y, globalPos.z);
}
}It's fairly specific to this engine I'm working on tho, e.g. you may ask what are labels I'd provide the original model file but sadly it's in a custom format used by this engine so I'd have to provide quite a few more classes. Anyway the issue was 2) and it all works perfectly now |
That's what my remark about 'that as a static initializer into each class that uses JOML, which is... meh. However, good to hear that it is resolved now. I also found If you have any ideas or thoughts about possible extensions or improvements of the builder package, just let me know. There are quite a few things in JglTF that I'm not satisfied with (and can no longer be changed), but I hope that I find some time to gradually improve it and reduce the pain-points based on the feedback from people who are actually using it. |
|
ah yeah using I'm not sure about improvements, i feel like it's already really easy to use, one would be the I wanted to ask if you could provide an example for reading animation data, clearly the way I'm doing it now is not correct import de.javagl.jgltf.model.*;
import de.javagl.jgltf.model.io.GltfModelReader;
import editor.cache.skeletal.math.Quaternionf;
import editor.cache.skeletal.math.Vector3f;
import java.io.IOException;
import java.nio.file.Path;
import java.util.List;
public class GltfReadTests {
public static void main(String[] args) throws IOException {
GltfModelReader reader = new GltfModelReader();
GltfModel model = reader.read(Path.of("./savedgltfmodels/simpleskin.gltf"));
List<AnimationModel> animationModels = model.getAnimationModels();
AnimationModel animation = animationModels.getFirst();
for (AnimationModel.Channel channel : animation.getChannels()) {
AnimationModel.Sampler sampler = channel.getSampler();
AccessorModel input = sampler.getInput();
AccessorModel output = sampler.getOutput();
AccessorFloatData inputData = (AccessorFloatData) input.getAccessorData();
AccessorFloatData outputData = (AccessorFloatData) output.getAccessorData();
for (int i = 0; i < inputData.getNumElements(); i++) {
float t = inputData.get(i, 0);
System.out.println(STR."t = \{t}");
for (int o = 0; o < outputData.getNumElements(); o++) {
switch (channel.getPath()) {
case "rotation" -> {
float x = outputData.get(o, 0);
float y = outputData.get(o, 1);
float z = outputData.get(o, 2);
float w = outputData.get(o, 3);
System.out.println(STR."Rotation: \{new Quaternionf(x, y, z, w)}");
}
case "translation" -> {
float x = outputData.get(o, 0);
float y = outputData.get(o, 1);
float z = outputData.get(o, 2);
System.out.println(STR."Translation: \{new Vector3f(x, y, z)}");
}
case "scale" -> {
float x = outputData.get(o, 0);
float y = outputData.get(o, 1);
float z = outputData.get(o, 2);
System.out.println(STR."Scale: \{new Vector3f(x, y, z)}");
}
}
}
System.out.println("-".repeat(100));
}
}
}
}This requires Java 21 to run however (if needed i can adjust it to work with java 8/11) This code clearly just prints all the same values for each t which is not what i'd want, i can see why the code does that tho, I'm just not sure how to adjust it so it reads it the way you'd expect Edit: i believe i got it right now, correct me if i made any mistakes please but the data looks correct (z and w seems flipped compared to what's in the GenerateSimpleSkin file however) public static void main(String[] args) throws IOException {
GltfModelReader reader = new GltfModelReader();
GltfModel model = reader.read(Path.of("./savedgltfmodels/simpleskin.gltf"));
List<AnimationModel> animationModels = model.getAnimationModels();
AnimationModel animation = animationModels.getFirst();
for (AnimationModel.Channel channel : animation.getChannels()) {
AnimationModel.Sampler sampler = channel.getSampler();
AccessorModel input = sampler.getInput();
AccessorModel output = sampler.getOutput();
AccessorFloatData inputData = (AccessorFloatData) input.getAccessorData();
AccessorFloatData outputData = (AccessorFloatData) output.getAccessorData();
// i guess inputData.getNumElements() should always be equal to outputData.getNumElements() anyway?
for (int i = 0; i < outputData.getNumElements(); i++) {
float t = inputData.get(i, 0);
System.out.println(STR."t = \{t}");
switch (channel.getPath()) {
case "rotation" -> {
float x = outputData.get(i, 0);
float y = outputData.get(i, 1);
float z = outputData.get(i, 2);
float w = outputData.get(i, 3);
System.out.println(STR."Rotation: \{new Quaternionf(x, y, z, w)}");
}
case "translation" -> {
float x = outputData.get(i, 0);
float y = outputData.get(i, 1);
float z = outputData.get(i, 2);
System.out.println(STR."Translation: \{new Vector3f(x, y, z)}");
}
case "scale" -> {
float x = outputData.get(i, 0);
float y = outputData.get(i, 1);
float z = outputData.get(i, 2);
System.out.println(STR."Scale: \{new Vector3f(x, y, z)}");
}
}
System.out.println("-".repeat(100));
}
}
}
}Edit: actually for my own mesh, i created an animation in blender consisting of 3 keyframes, however Edit 2: nevermind, there was just multiple |
|
Animations are one of the next things that I'll tackle in the "builder" package. There are probably some low-hanging fruits, but ... the devil is in the detail. The updated method looks correct. As an example for the
And it has two methods for accessing the data, namely,
So when you're calling
That's very surprising for me (and I'll probably try that out and validate it locally). But I have to admit that there are some complexities in the Blender animation functionality (and correspondingly, some configuration options and caveats for exporting these animations to glTF) that I don't have on the radar right now. |
|
Thanks for the explanation Yeah blender/glTF exporting is somewhat weird.. displaying the model in an online glTF viewer shows that it has 3 animations, the first one does nothing, the 2nd and 3rd ones are identical After creating an animation again (similar one) it exported just fine (1 animation) but o well this is probably just some blender glTF exporter related thing |
|
I'll have to check that |
|
Yeah that was actually it the interpolation mode was |
|
@Suicolen You may have seen that there is an open PR for the model builder package at #103 . (It has, once more, been open for too long - I've been distracted with other stuff). I'll probably try to wrap this up and merge it soon. Maybe you want to have a look, maybe try it out, and provide other feedback. (I considered to start adding something like an |
|
@javagl I will try it out asap (currently working on some other things i need to finish first before continuing with my glTF related project) as for |
Hi, it's been close to 2 years since i used this library but I'd like to once again attempt exporting my own custom model format to glTF (and also vice versa, but for now exporting is more important) last time i managed to successfully do it but it was quite difficult to do so, internal classes like
BufferStructureBuilderdid make it quite a bit more bearable howeverI remember jgltf model builder being mentioned in my last issue: #78 however i decided to put a hold on the project at that time as some other things came up.
My question being, is it still actively being worked on and a lot of it might change or is it mostly finished and i should probably use it?
There's this note: This library is still subject to change.
But if there won't be any major changes then i guess i could use it.
And are there any examples of it being used, if not I'd very much appreciate if some simple examples could be made, preferably of a skinned mesh, because in my case i need to export the mesh, the armature and potentially also animations, altho the latter isn't that important and i mainly need to export the mesh and the armature for now.
The text was updated successfully, but these errors were encountered: