Importing and exporting SkinnedMeshRenderer with blendshapes (#363)

* exporting smr and blendshapes * import gltf mesh with blendshapes/morph targets * add the ability to control the final LOD when far away (cull vs render last lod) (#364) * add the ability to control whether objects with lod should render when very far away * pr feedback * Improve the performance of skinned animation import (#360) * Improve the performance of skinned animation import Current implementation of importing animation curve is very slow for animations with lots of key frames. (e.g. motion capture animation ). The bottleneck is in AnimationCurve.keys return a copy instead of reference. Hence, updating key frame in AnimationCurve is expensive. I build a list of keyframe for the curve, calculate the tangents from interpolation mode and then feed the key frames to the curve to avoid the overhead. * Improve the performance of skinned animation import 1, put the delegate function at the top of the file 2, replace spaces with tab 3, replace the keyframe list cache with 2d array * fix level 3 warning. move tests to get the nunit dependency in editor (#365) * move tests * fixes level 3 warnings * remove redundant namespace * fix ups to level 3 warnings. remove unnecessary namespaces * Fix mapping between GLTF sampler and Unity filter/wrap modes (#375) * A fix for the freeze in editor when importing GLB-files (might affect gltf too), by never using multithreading in editor (#368) * IsMultithreaded is now a property that always returns false when Application.isEditor is true. This prevents a "hard-freeze" in Unity editor when importing any GLB. * Update GLTFSceneImporter.cs Fixed a comment * Added conditional checks in SetUpBlendShapes to see if TargetNames and Semantic properties POSITION, NORMAL, TANGENT exist * Expose GPU-only texture option to user and fix mismatching sampler st… (#378) * Expose GPU-only texture option to user and fix mismatching sampler state warning 'if' condition * Make linear/sRGB color space setting match spec * Update KHR_texture_transform support (#371) * Update KHR_texture_transform support * Flip tex offset y coord * Fix non-conformant texture origin * Yield timeout should be common to all loaders as framerate is an appl… (#379) * Yield timeout should be common to all loaders as framerate is an application-wide concern * Resolve comments * . * . * Create UPM compliant packages in releases (#385) * Enable UPM packages to be generated in releases * Travis OSX package should already contain zip package * Remove draft=true for releases deployment * Remove logging of private data (#386) * Support iOS for GLTFSerialization.dll (#387) * Prevent InvalidCastException (#399) Related to Issue #398 * Fix stream disposing bug (#388) * Invert the _Glossiness Material value (#383) * The comparison of matrix class fails due to using the operator instead of .Equal causing both matrix and RST data (#392) * Add Roughness float values to MetalRoughMap.cs * Add names to textures created by GLTFSceneImporter * Fix texture memory leak * Add task cancellation support * Add image name to texture name * Update UnityGLTF/Assets/UnityGLTF/Scripts/Cache/RefCountedCacheData.cs Co-Authored-By: rferrese <[email protected]> * Change cancellation in ConstructMesh * Camera in test scenes now moves only when dragged (mousedown) and the movement math is cleaner and more accurate, and zoom is now exponential instead of linear. (#413) Also clean up some warnings around unused variables. * Fix loading time issue (#414) * Simplify AsyncCoroutineHelper * revert default budget * Add missing meta file for integration test directory (#419) * Add a component to display the list of models from gltf-Sample-Models and let you load them (#420) * Add a component to display the list of sample models and let you load them at runtime. * Some improvements to the model list loader and adding a main scene to host it * Fix warning suppression * Address pr feedback. Remove commented line. Give a nice error if the model list json fails to download. Also improve orbit camera so that scrolling the other list doesn't cause it to zoom. * OrbitCamera now supports right mouse buttons to change position (#422) * right mouse camera controls * More accurate right click mouse motion * Set the correct filter mode when loading texture samplers (#421) * Fix for loading performance issue (#423) * Set timeout after successfully awaiting. * fix spaces and remove file that unity keeps deleting * Fix UPM package generation issues for UWP (#406) * Include .asmdef files for correct UPM package generation * Fix missing Integrations .meta file position * Splitting out UnityGLTFEditor scripts with its own asmdef file * Build script updates to properly support UWP with UPM compliant package * Include .asmdef files for correct UPM package generation * Fix missing Integrations .meta file position * Splitting out UnityGLTFEditor scripts with its own asmdef file * Build script updates to properly support UWP with UPM compliant package * Fixing Travis build environment issues * Include .asmdef files for correct UPM package generation * Fix missing Integrations .meta file position * Splitting out UnityGLTFEditor scripts with its own asmdef file * Build script updates to properly support UWP with UPM compliant package * Fixing Travis build environment issues * Pull down upstream changes * Fixing merge conflict * Add Roughness float values to MetalRoughMap.cs * Add names to textures created by GLTFSceneImporter * Fix texture memory leak * Add task cancellation support * Add image name to texture name * Update UnityGLTF/Assets/UnityGLTF/Scripts/Cache/RefCountedCacheData.cs Co-Authored-By: rferrese <[email protected]> * Change cancellation in ConstructMesh * Camera in test scenes now moves only when dragged (mousedown) and the movement math is cleaner and more accurate, and zoom is now exponential instead of linear. (#413) Also clean up some warnings around unused variables. * Fix loading time issue (#414) * Simplify AsyncCoroutineHelper * revert default budget * Add missing meta file for integration test directory (#419) * Add a component to display the list of models from gltf-Sample-Models and let you load them (#420) * Add a component to display the list of sample models and let you load them at runtime. * Some improvements to the model list loader and adding a main scene to host it * Fix warning suppression * Address pr feedback. Remove commented line. Give a nice error if the model list json fails to download. Also improve orbit camera so that scrolling the other list doesn't cause it to zoom. * OrbitCamera now supports right mouse buttons to change position (#422) * right mouse camera controls * More accurate right click mouse motion * Set the correct filter mode when loading texture samplers (#421) * Include .asmdef files for correct UPM package generation * Fix missing Integrations .meta file position * Splitting out UnityGLTFEditor scripts with its own asmdef file * Build script updates to properly support UWP with UPM compliant package * Include .asmdef files for correct UPM package generation * Splitting out UnityGLTFEditor scripts with its own asmdef file * Fixing Travis build environment issues * Include .asmdef files for correct UPM package generation * Splitting out UnityGLTFEditor scripts with its own asmdef file * Pull down upstream changes * Fixing merge conflict * Enable Azure Pipelines UPM releases (#2) * Update Export-upm-packages.sh to support Azure Pipelines * Remove UPM Package generation from .travis.yml UPM package generation now works with Azure Pipelines * Add ImportProgress indication (#424) * Add ImportProgress indication * PR Comments * Make all of the maps public so that callers can derive from them. Previously they were internal and it worked in some Unity projects because they'd all end up in the same assembly. Now that we've added assemblydefs, the UnityGLTF classes are in their own assembly. (#426) * Fix uwp build (#427) * Update GLTFSerialization/GLTFSerialization/Schema/MeshPrimitive.cs remove whitespace Co-Authored-By: Adam Mitchell <[email protected]> * remove whitespace Co-Authored-By: Adam Mitchell <[email protected]> * remove whitespace Co-Authored-By: Adam Mitchell <[email protected]> * rename bsname to blendShapeName * add comment clarifying that GLTF weights are [0, 1] range but Unity weights are [0, 100] range Co-Authored-By: Adam Mitchell <[email protected]> * change logerror to logwarning, added some clarity of what will happen (using minimum blendshape count) * add curly braces to if else statement * remove space Co-Authored-By: Adam Mitchell <[email protected]> * remove whitespace Co-Authored-By: Adam Mitchell <[email protected]> * add function ContainsValidRenderer for better readability * remove whitespace Co-Authored-By: Adam Mitchell <[email protected]>
KhronosGroup · Jun 10, 2019 · b39a454 · b39a454
1 parent 7299431
commit b39a454
Show file tree

Hide file tree

Showing 5 changed files with 308 additions and 25 deletions.
diff --git a/GLTFSerialization/GLTFSerialization/GLTFHelpers.cs b/GLTFSerialization/GLTFSerialization/GLTFHelpers.cs
@@ -320,6 +320,40 @@ public static void BuildMeshAttributes(ref Dictionary<string, AttributeAccessor>
 			}
 		}
 
+		public static void BuildTargetAttributes (ref Dictionary<string, AttributeAccessor> attributes)
+		{
+			if (attributes.ContainsKey(SemanticProperties.POSITION))
+			{
+				var attributeAccessor = attributes[SemanticProperties.POSITION];
+				NumericArray resultArray = attributeAccessor.AccessorContent;
+				byte[] bufferViewCache;
+				uint offset = LoadBufferView(attributeAccessor, out bufferViewCache);
+				attributeAccessor.AccessorId.Value.AsVector3Array(ref resultArray, bufferViewCache, offset);
+				attributeAccessor.AccessorContent = resultArray;
+			}
+
+			if (attributes.ContainsKey(SemanticProperties.NORMAL))
+			{
+				var attributeAccessor = attributes[SemanticProperties.NORMAL];
+				NumericArray resultArray = attributeAccessor.AccessorContent;
+				byte[] bufferViewCache;
+				uint offset = LoadBufferView(attributeAccessor, out bufferViewCache);
+				attributeAccessor.AccessorId.Value.AsVector3Array(ref resultArray, bufferViewCache, offset);
+				attributeAccessor.AccessorContent = resultArray;
+			}
+
+			if (attributes.ContainsKey(SemanticProperties.TANGENT))
+			{
+				var attributeAccessor = attributes[SemanticProperties.TANGENT];
+				NumericArray resultArray = attributeAccessor.AccessorContent;
+				byte[] bufferViewCache;
+				uint offset = LoadBufferView(attributeAccessor, out bufferViewCache);
+				attributeAccessor.AccessorId.Value.AsVector3Array(ref resultArray, bufferViewCache, offset);
+				attributeAccessor.AccessorContent = resultArray;
+			}
+
+		}
+
 		public static void BuildBindPoseSamplers(ref AttributeAccessor attributeAccessor)
 		{
 			NumericArray resultArray = attributeAccessor.AccessorContent;

diff --git a/GLTFSerialization/GLTFSerialization/Schema/MeshPrimitive.cs b/GLTFSerialization/GLTFSerialization/Schema/MeshPrimitive.cs
@@ -47,6 +47,8 @@ public class MeshPrimitive : GLTFProperty
 		/// TODO: Make dictionary key enums?
 		public List<Dictionary<string, AccessorId>> Targets;
 
+		public List<string> TargetNames;
+
 		public MeshPrimitive()
 		{
 		}
@@ -89,6 +91,11 @@ public MeshPrimitive(MeshPrimitive meshPrimitive, GLTFRoot gltfRoot) : base(mesh
 					Targets.Add(target);
 				}
 			}
+
+			if (meshPrimitive.TargetNames != null)
+			{
+				TargetNames = new List<string>(meshPrimitive.TargetNames);
+			}
 		}
 
 		public static int[] GenerateTriangles(int vertCount)
@@ -211,6 +218,27 @@ public static MeshPrimitive Deserialize(GLTFRoot root, JsonReader reader)
 							},
 							skipStartObjectRead: true);
 						});
+						break;
+					case "extras":
+						// GLTF does not support morph target names, serialize in extras for now
+						// https://github.com/KhronosGroup/glTF/issues/1036
+						if (reader.Read() && reader.TokenType == JsonToken.StartObject)
+						{
+							while (reader.Read() && reader.TokenType == JsonToken.PropertyName)
+							{
+								var extraProperty = reader.Value.ToString();
+								switch (extraProperty)
+								{
+									case "targetNames":
+										primitive.TargetNames = reader.ReadStringList();
+										break;
+
+								}
+
+							}
+
+						}
+
 						break;
 					default:
 						primitive.DefaultPropertyDeserializer(root, reader);
@@ -271,6 +299,22 @@ public override void Serialize(JsonWriter writer)
 				writer.WriteEndArray();
 			}
 
+			// GLTF does not support morph target names, serialize in extras for now
+			// https://github.com/KhronosGroup/glTF/issues/1036
+			if (TargetNames != null && TargetNames.Count > 0)
+			{
+				writer.WritePropertyName("extras");
+				writer.WriteStartObject();
+				writer.WritePropertyName("targetNames");
+				writer.WriteStartArray();
+				foreach (var targetName in TargetNames)
+				{
+					writer.WriteValue(targetName);
+				}
+				writer.WriteEndArray();
+				writer.WriteEndObject();
+			}
+
 			base.Serialize(writer);
 
 			writer.WriteEndObject();

diff --git a/UnityGLTF/Assets/UnityGLTF/Scripts/Cache/MeshCacheData.cs b/UnityGLTF/Assets/UnityGLTF/Scripts/Cache/MeshCacheData.cs
@@ -10,6 +10,8 @@ public class MeshCacheData : IDisposable
 		public Mesh LoadedMesh { get; set; }
 		public Dictionary<string, AttributeAccessor> MeshAttributes { get; set; }
 		public GameObject PrimitiveGO { get; set; }
+
+		public List<Dictionary<string, AttributeAccessor>> Targets;
 
 		public MeshCacheData()
 		{

diff --git a/UnityGLTF/Assets/UnityGLTF/Scripts/GLTFSceneExporter.cs b/UnityGLTF/Assets/UnityGLTF/Scripts/GLTFSceneExporter.cs
@@ -487,9 +487,17 @@ private NodeId ExportNode(Transform nodeTransform)
 				// associate unity meshes with gltf mesh id
 				foreach (var prim in primitives)
 				{
-					var filter = prim.GetComponent<MeshFilter>();
-					var renderer = prim.GetComponent<MeshRenderer>();
-					_primOwner[new PrimKey { Mesh = filter.sharedMesh, Material = renderer.sharedMaterial }] = node.Mesh;
+					var smr = prim.GetComponent<SkinnedMeshRenderer>();
+					if (smr != null)
+					{
+						_primOwner[new PrimKey { Mesh = smr.sharedMesh, Material = smr.sharedMaterial }] = node.Mesh;
+					}
+					else
+					{
+						var filter = prim.GetComponent<MeshFilter>();
+						var renderer = prim.GetComponent<MeshRenderer>();
+						_primOwner[new PrimKey { Mesh = filter.sharedMesh, Material = renderer.sharedMaterial }] = node.Mesh;
+					}
 				}
 			}
 
@@ -568,19 +576,26 @@ private CameraId ExportCamera(Camera unityCamera)
 			return id;
 		}
 
+		private static bool ContainsValidRenderer (GameObject gameObject)
+		{
+			return (gameObject.GetComponent<MeshFilter>() != null && gameObject.GetComponent<MeshRenderer>() != null) 
+					|| (gameObject.GetComponent<SkinnedMeshRenderer>() != null);
+		}
+
 		private void FilterPrimitives(Transform transform, out GameObject[] primitives, out GameObject[] nonPrimitives)
 		{
 			var childCount = transform.childCount;
 			var prims = new List<GameObject>(childCount + 1);
 			var nonPrims = new List<GameObject>(childCount);
 
 			// add another primitive if the root object also has a mesh
-			if (transform.gameObject.GetComponent<MeshFilter>() != null
-				&& transform.gameObject.GetComponent<MeshRenderer>() != null)
+			if (transform.gameObject.activeSelf)
 			{
-				prims.Add(transform.gameObject);
+				if (ContainsValidRenderer(transform.gameObject))
+				{
+					prims.Add(transform.gameObject);
+				}
 			}
-
 			for (var i = 0; i < childCount; i++)
 			{
 				var go = transform.GetChild(i).gameObject;
@@ -600,14 +615,14 @@ private static bool IsPrimitive(GameObject gameObject)
 			 * Primitives have the following properties:
 			 * - have no children
 			 * - have no non-default local transform properties
-			 * - have MeshFilter and MeshRenderer components
+			 * - have MeshFilter and MeshRenderer components OR has SkinnedMeshRenderer component
 			 */
 			return gameObject.transform.childCount == 0
 				&& gameObject.transform.localPosition == Vector3.zero
 				&& gameObject.transform.localRotation == Quaternion.identity
 				&& gameObject.transform.localScale == Vector3.one
-				&& gameObject.GetComponent<MeshFilter>() != null
-				&& gameObject.GetComponent<MeshRenderer>() != null;
+				&& ContainsValidRenderer(gameObject);
+
 		}
 
 		private MeshId ExportMesh(string name, GameObject[] primitives)
@@ -617,10 +632,19 @@ private MeshId ExportMesh(string name, GameObject[] primitives)
 			var key = new PrimKey();
 			foreach (var prim in primitives)
 			{
-				var filter = prim.GetComponent<MeshFilter>();
-				var renderer = prim.GetComponent<MeshRenderer>();
-				key.Mesh = filter.sharedMesh;
-				key.Material = renderer.sharedMaterial;
+				var smr = prim.GetComponent<SkinnedMeshRenderer>();
+				if (smr != null)
+				{
+					key.Mesh = smr.sharedMesh;
+					key.Material = smr.sharedMaterial;
+				}
+				else
+				{
+					var filter = prim.GetComponent<MeshFilter>();
+					var renderer = prim.GetComponent<MeshRenderer>();
+					key.Mesh = filter.sharedMesh;
+					key.Material = renderer.sharedMaterial;
+				}
 
 				MeshId tempMeshId;
 				if (_primOwner.TryGetValue(key, out tempMeshId) && (existingMeshId == null || tempMeshId == existingMeshId))
@@ -651,13 +675,13 @@ private MeshId ExportMesh(string name, GameObject[] primitives)
 			mesh.Primitives = new List<MeshPrimitive>(primitives.Length);
 			foreach (var prim in primitives)
 			{
-				MeshPrimitive[] meshPrimitives = ExportPrimitive(prim);
+				MeshPrimitive[] meshPrimitives = ExportPrimitive(prim, mesh);
 				if (meshPrimitives != null)
 				{
 					mesh.Primitives.AddRange(meshPrimitives);
 				}
 			}
-
+			
 			var id = new MeshId
 			{
 				Id = _root.Meshes.Count,
@@ -669,18 +693,28 @@ private MeshId ExportMesh(string name, GameObject[] primitives)
 		}
 
 		// a mesh *might* decode to multiple prims if there are submeshes
-		private MeshPrimitive[] ExportPrimitive(GameObject gameObject)
+		private MeshPrimitive[] ExportPrimitive(GameObject gameObject, GLTFMesh mesh)
 		{
+			Mesh meshObj = null;
+			SkinnedMeshRenderer smr = null;
 			var filter = gameObject.GetComponent<MeshFilter>();
-			var meshObj = filter.sharedMesh;
+			if (filter != null)
+			{
+				meshObj = filter.sharedMesh;
+			}
+			else
+			{
+				smr = gameObject.GetComponent<SkinnedMeshRenderer>();
+				meshObj = smr.sharedMesh;
+			}
 			if (meshObj == null)
 			{
 				Debug.LogError(string.Format("MeshFilter.sharedMesh on gameobject:{0} is missing , skipping", gameObject.name));
 				return null;
 			}
 
 			var renderer = gameObject.GetComponent<MeshRenderer>();
-			var materialsObj = renderer.sharedMaterials;
+			var materialsObj = renderer != null ? renderer.sharedMaterials : smr.sharedMaterials;
 
 			var prims = new MeshPrimitive[meshObj.subMeshCount];
 
@@ -702,7 +736,7 @@ private MeshPrimitive[] ExportPrimitive(GameObject gameObject)
 
 			AccessorId aPosition = null, aNormal = null, aTangent = null,
 				aTexcoord0 = null, aTexcoord1 = null, aColor0 = null;
-
+				
 			aPosition = ExportAccessor(SchemaExtensions.ConvertVector3CoordinateSpaceAndCopy(meshObj.vertices, SchemaExtensions.CoordinateSpaceConversionScale));
 
 			if (meshObj.normals.Length != 0)
@@ -753,6 +787,8 @@ private MeshPrimitive[] ExportPrimitive(GameObject gameObject)
 					primitive.Material = lastMaterialId;
 				}
 
+				ExportBlendShapes(smr, meshObj, primitive, mesh);
+
 				prims[submesh] = primitive;
 			}
 
@@ -796,7 +832,7 @@ private MaterialId ExportMaterial(Material materialObj)
 
 			material.DoubleSided = materialObj.HasProperty("_Cull") &&
 				materialObj.GetInt("_Cull") == (float)CullMode.Off;
-
+				
 			if (materialObj.HasProperty("_EmissionColor"))
 			{
 				material.EmissiveFactor = materialObj.GetColor("_EmissionColor").ToNumericsColorRaw();
@@ -878,6 +914,56 @@ private MaterialId ExportMaterial(Material materialObj)
 			return id;
 		}
 
+		// Blend Shapes / Morph Targets
+		// Adopted from Gary Hsu (bghgary)
+		// https://github.com/bghgary/glTF-Tools-for-Unity/blob/master/UnityProject/Assets/Gltf/Editor/Exporter.cs
+		private void ExportBlendShapes(SkinnedMeshRenderer smr, Mesh meshObj, MeshPrimitive primitive, GLTFMesh mesh)
+		{
+			if (smr != null && meshObj.blendShapeCount > 0)
+			{
+				List<Dictionary<string, AccessorId>> targets = new List<Dictionary<string, AccessorId>>(meshObj.blendShapeCount);
+				List<Double> weights = new List<double>(meshObj.blendShapeCount);
+				List<string> targetNames = new List<string>(meshObj.blendShapeCount);
+
+				for (int blendShapeIndex = 0; blendShapeIndex < meshObj.blendShapeCount; blendShapeIndex++)
+				{
+
+					targetNames.Add(meshObj.GetBlendShapeName(blendShapeIndex));
+					// As described above, a blend shape can have multiple frames.  Given that glTF only supports a single frame
+					// per blend shape, we'll always use the final frame (the one that would be for when 100% weight is applied).
+					int frameIndex = meshObj.GetBlendShapeFrameCount(blendShapeIndex) - 1;
+
+					var deltaVertices = new Vector3[meshObj.vertexCount];
+					var deltaNormals = new Vector3[meshObj.vertexCount];
+					var deltaTangents = new Vector3[meshObj.vertexCount];
+					meshObj.GetBlendShapeFrameVertices(blendShapeIndex, frameIndex, deltaVertices, deltaNormals, deltaTangents);
+
+					targets.Add(new Dictionary<string, AccessorId>
+						{
+							{ SemanticProperties.POSITION, ExportAccessor(SchemaExtensions.ConvertVector3CoordinateSpaceAndCopy( deltaVertices, SchemaExtensions.CoordinateSpaceConversionScale)) },
+							{ SemanticProperties.NORMAL, ExportAccessor(SchemaExtensions.ConvertVector3CoordinateSpaceAndCopy(deltaNormals,SchemaExtensions.CoordinateSpaceConversionScale))},
+							{ SemanticProperties.TANGENT, ExportAccessor(SchemaExtensions.ConvertVector3CoordinateSpaceAndCopy(deltaTangents, SchemaExtensions.CoordinateSpaceConversionScale)) },
+						});
+
+					// We need to get the weight from the SkinnedMeshRenderer because this represents the currently
+					// defined weight by the user to apply to this blend shape.  If we instead got the value from
+					// the unityMesh, it would be a _per frame_ weight, and for a single-frame blend shape, that would
+					// always be 100.  A blend shape might have more than one frame if a user wanted to more tightly
+					// control how a blend shape will be animated during weight changes (e.g. maybe they want changes
+					// between 0-50% to be really minor, but between 50-100 to be extreme, hence they'd have two frames
+					// where the first frame would have a weight of 50 (meaning any weight between 0-50 should be relative
+					// to the values in this frame) and then any weight between 50-100 would be relevant to the weights in
+					// the second frame.  See Post 20 for more info:
+					// https://forum.unity3d.com/threads/is-there-some-method-to-add-blendshape-in-editor.298002/#post-2015679
+					weights.Add(smr.GetBlendShapeWeight(blendShapeIndex) / 100);
+				}
+
+				mesh.Weights = weights;
+				primitive.Targets = targets;
+				primitive.TargetNames = targetNames;
+			}
+		}
+
 		private bool IsPBRMetallicRoughness(Material material)
 		{
 			return material.HasProperty("_Metallic") && material.HasProperty("_MetallicGlossMap");
@@ -1168,7 +1254,7 @@ private ImageId ExportImage(Texture texture, TextureMapType texturMapType)
 				textureMapType = texturMapType
 			});
 
-			var imagePath = _retrieveTexturePathDelegate(texture);
+			var imagePath =_retrieveTexturePathDelegate(texture);
 			if (string.IsNullOrEmpty(imagePath))
 			{
 				imagePath = texture.name;