Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add basic memory statistics in ModelExperimental #10397

Merged
merged 59 commits into from
Jun 7, 2022
Merged
Show file tree
Hide file tree
Changes from 54 commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
861f730
Add model statistics object
ptrgags May 17, 2022
55e8a4f
Add statistics for the geometry stage
ptrgags May 17, 2022
0f2b8e8
Add a unique ID to buffers to avoid double-counting
ptrgags May 18, 2022
ecf9663
Compute memory size for old batch tables
ptrgags May 19, 2022
dc0d855
Add a separate category for metadata
ptrgags May 19, 2022
bbfeb37
Add memory size to PropertyTable
ptrgags May 19, 2022
7ce9e0f
Track statistics in more stages
ptrgags May 19, 2022
8cb78ac
Add notes of things to change
ptrgags May 19, 2022
470ebd2
Check for corner case in processing buffers
ptrgags May 20, 2022
a11a94e
Update other pipeline stages
ptrgags May 20, 2022
da1f9d4
Get ModelExperimental statistics displaying
ptrgags May 20, 2022
99504e9
remove metadataSizeInBytes
ptrgags May 20, 2022
015c3d0
Update unit tests to use a statistics object
ptrgags May 20, 2022
c4b2171
Load original indices in the loader, not the pipeline
ptrgags May 23, 2022
6bd8b43
Update more unit tests
ptrgags May 23, 2022
ad6873c
Continue fixing unit tests
ptrgags May 23, 2022
9763365
Remove metadataByteLength
ptrgags May 24, 2022
f533cb2
Standardize on byteLength
ptrgags May 24, 2022
f10d68a
It wasn't dead code after all...
ptrgags May 24, 2022
8705cec
Add an if-guard around resetting the draw commands
ptrgags May 24, 2022
96a8974
Clone unit test data to avoid test failures
ptrgags May 24, 2022
0c4c668
Update tests for metadata table statistics
ptrgags May 24, 2022
a1a8fe8
Test structural metadata byte length
ptrgags May 24, 2022
908ed45
Merge branch 'main' into model-experimental-statistics
ptrgags May 24, 2022
363038a
Add specs for ModelExperimentalStatistics
ptrgags May 24, 2022
05e307d
Update specs for WireframePipelineStage
ptrgags May 24, 2022
42f1f3a
Merge branch 'main' into model-experimental-statistics
ptrgags May 25, 2022
2691456
Fix GeometryPipelineStage tests
ptrgags May 25, 2022
e691b85
Update instancing stage unit tests
ptrgags May 25, 2022
d6d4646
Test statistics for the material stage
ptrgags May 25, 2022
df21144
Update metadata pipeline stage tests
ptrgags May 25, 2022
506bb42
Update morph target stage
ptrgags May 25, 2022
dc8597a
Update picking stage tests
ptrgags May 25, 2022
a463878
Update feature ID pipeline stage
ptrgags May 25, 2022
5b48e2b
Update specs for geometry pipeline stage
ptrgags May 26, 2022
7814bef
Merge branch 'main' into model-experimental-statistics
ptrgags May 26, 2022
f9864a0
+= not =
ptrgags May 26, 2022
d4cd90c
PR feedback
ptrgags Jun 1, 2022
6e214be
Move material statistics to its own function
ptrgags Jun 1, 2022
025f832
Attempt to make a statistics update stage
ptrgags Jun 1, 2022
528ab26
Make the statistics stage an update stage
ptrgags Jun 1, 2022
6c788b3
Remove statistics dirty flag
ptrgags Jun 1, 2022
d36e783
Update ModelExperimentalPrimitive specs
ptrgags Jun 1, 2022
90a879a
Move geometry statistics tests to StatisticsPipelineStage
ptrgags Jun 1, 2022
7cfb076
Move material texture specs
ptrgags Jun 1, 2022
7d2a3df
Move morph target memory specs to statistics stage specs
ptrgags Jun 1, 2022
aae66a9
Move metadata statistics tests
ptrgags Jun 1, 2022
c2d89f9
Sweep through other pipeline stages
ptrgags Jun 1, 2022
9c154d2
Add feature ID textures to the statistics stage
ptrgags Jun 1, 2022
888e338
Finish adding tests for StatisticsPipelineStage
ptrgags Jun 6, 2022
bab3c6d
Merge branch 'main' into model-experimental-statistics
ptrgags Jun 6, 2022
e760e3b
Add a test for restting draw commands after loading
ptrgags Jun 6, 2022
82efe86
Remove outdated statistics checks from the 2D stage
ptrgags Jun 6, 2022
ca9607b
Remove flaky test
ptrgags Jun 6, 2022
4bf895d
PR Feedback
ptrgags Jun 7, 2022
a8c9f7a
Test default materials
ptrgags Jun 7, 2022
888af9d
Update CHANGES.md
ptrgags Jun 7, 2022
832de63
Fix typo, remove console log in unit test
Jun 7, 2022
9ec6d24
Run prettier
Jun 7, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Source/Renderer/Buffer.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import Check from "../Core/Check.js";
import createGuid from "../Core/createGuid.js";
import defaultValue from "../Core/defaultValue.js";
import defined from "../Core/defined.js";
import destroyObject from "../Core/destroyObject.js";
Expand Down Expand Up @@ -61,6 +62,7 @@ function Buffer(options) {
gl.bufferData(bufferTarget, hasArray ? typedArray : sizeInBytes, usage);
gl.bindBuffer(bufferTarget, null);

this._id = createGuid();
this._gl = gl;
this._webgl2 = options.context._webgl2;
this._bufferTarget = bufferTarget;
Expand Down
31 changes: 31 additions & 0 deletions Source/Scene/BatchTableHierarchy.js
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ export default function BatchTableHierarchy(options) {
this._parentIndexes = undefined;
this._parentIds = undefined;

// Total memory used by the typed arrays
this._byteLength = 0;

//>>includeStart('debug', pragmas.debug);
Check.typeOf.object("options.extension", options.extension);
//>>includeEnd('debug');
Expand All @@ -41,6 +44,14 @@ export default function BatchTableHierarchy(options) {
//>>includeEnd('debug');
}

Object.defineProperties(BatchTableHierarchy.prototype, {
byteLength: {
get: function () {
return this._byteLength;
},
},
});

/**
* Parse the batch table hierarchy from the
* <code>3DTILES_batch_table_hierarchy</code> extension.
Expand All @@ -61,6 +72,7 @@ function initialize(hierarchy, hierarchyJson, binaryBody) {
let parentCounts = hierarchyJson.parentCounts;
let parentIds = hierarchyJson.parentIds;
let parentIdsLength = instancesLength;
let byteLength = 0;

if (defined(classIds.byteOffset)) {
classIds.componentType = defaultValue(
Expand All @@ -74,6 +86,7 @@ function initialize(hierarchy, hierarchyJson, binaryBody) {
binaryBody.byteOffset + classIds.byteOffset,
instancesLength
);
byteLength += classIds.byteLength;
}

let parentIndexes;
Expand All @@ -90,13 +103,16 @@ function initialize(hierarchy, hierarchyJson, binaryBody) {
binaryBody.byteOffset + parentCounts.byteOffset,
instancesLength
);
byteLength += parentCounts.byteLength;
}
parentIndexes = new Uint16Array(instancesLength);
parentIdsLength = 0;
for (i = 0; i < instancesLength; ++i) {
parentIndexes[i] = parentIdsLength;
parentIdsLength += parentCounts[i];
}

byteLength += parentIndexes.byteLength;
}

if (defined(parentIds) && defined(parentIds.byteOffset)) {
Expand All @@ -111,6 +127,8 @@ function initialize(hierarchy, hierarchyJson, binaryBody) {
binaryBody.byteOffset + parentIds.byteOffset,
parentIdsLength
);

byteLength += parentIds.byteLength;
}

const classesLength = classes.length;
Expand All @@ -122,6 +140,7 @@ function initialize(hierarchy, hierarchyJson, binaryBody) {
properties,
binaryBody
);
byteLength += countBinaryPropertyMemory(binaryProperties);
classes[i].instances = combine(binaryProperties, properties);
}

Expand All @@ -132,13 +151,15 @@ function initialize(hierarchy, hierarchyJson, binaryBody) {
classIndexes[i] = classCounts[classId];
++classCounts[classId];
}
byteLength += classIndexes.byteLength;

hierarchy._classes = classes;
hierarchy._classIds = classIds;
hierarchy._classIndexes = classIndexes;
hierarchy._parentCounts = parentCounts;
hierarchy._parentIndexes = parentIndexes;
hierarchy._parentIds = parentIds;
hierarchy._byteLength = byteLength;
}

function getBinaryProperties(featuresLength, properties, binaryBody) {
Expand Down Expand Up @@ -189,6 +210,16 @@ function getBinaryProperties(featuresLength, properties, binaryBody) {
return binaryProperties;
}

function countBinaryPropertyMemory(binaryProperties) {
let byteLength = 0;
for (const name in binaryProperties) {
if (binaryProperties.hasOwnProperty(name)) {
byteLength += binaryProperties[name].typedArray.byteLength;
}
}
return byteLength;
}

//>>includeStart('debug', pragmas.debug);
const scratchValidateStack = [];
function validateHierarchy(hierarchy) {
Expand Down
8 changes: 7 additions & 1 deletion Source/Scene/BatchTexture.js
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ Object.defineProperties(BatchTexture.prototype, {
* @readonly
* @private
*/
memorySizeInBytes: {
byteLength: {
get: function () {
let memory = 0;
if (defined(this._pickTexture)) {
Expand Down Expand Up @@ -475,6 +475,9 @@ function createPickTexture(batchTexture, context) {
}

batchTexture._pickTexture = createTexture(batchTexture, context, bytes);

// Make sure the tileset statistics are updated the frame when the pick
// texture is created.
if (defined(statistics)) {
statistics.batchTableByteLength += batchTexture._pickTexture.sizeInBytes;
}
Expand Down Expand Up @@ -510,6 +513,9 @@ BatchTexture.prototype.update = function (tileset, frameState) {
// Create batch texture on-demand
if (!defined(this._batchTexture)) {
this._batchTexture = createTexture(this, context, this._batchValues);

// Make sure the tileset statistics are updated the frame when the
// batch texture is created.
if (defined(this._statistics)) {
this._statistics.batchTableByteLength += this._batchTexture.sizeInBytes;
}
Expand Down
2 changes: 1 addition & 1 deletion Source/Scene/Batched3DModel3DTileContent.js
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ Object.defineProperties(Batched3DModel3DTileContent.prototype, {

batchTableByteLength: {
get: function () {
return this.batchTable.memorySizeInBytes;
return this.batchTable.batchTableByteLength;
},
},

Expand Down
42 changes: 39 additions & 3 deletions Source/Scene/Cesium3DTileBatchTable.js
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,16 @@ function Cesium3DTileBatchTable(
batchTableJson,
batchTableBinary
);
this._batchTableBinaryProperties = getBinaryProperties(

const binaryProperties = getBinaryProperties(
featuresLength,
properties,
batchTableBinary
);
this._binaryPropertiesByteLength = countBinaryPropertyMemory(
binaryProperties
);
this._batchTableBinaryProperties = binaryProperties;

this._content = content;

Expand All @@ -78,9 +83,26 @@ function Cesium3DTileBatchTable(
Cesium3DTileBatchTable._deprecationWarning = deprecationWarning;

Object.defineProperties(Cesium3DTileBatchTable.prototype, {
memorySizeInBytes: {
/**
* Size of the batch table, including the batch table hierarchy's binary
* buffers and any binary properties. JSON data is not counted.
*
* @memberof Cesium3DTileBatchTable.prototype
* @type {Number}
* @readonly
* @private
*/
batchTableByteLength: {
get: function () {
return this._batchTexture.memorySizeInBytes;
let totalByteLength = this._binaryPropertiesByteLength;

if (defined(this._batchTableHierarchy)) {
totalByteLength += this._batchTableHierarchy.byteLength;
}

totalByteLength += this._batchTexture.byteLength;

return totalByteLength;
},
},
});
Expand Down Expand Up @@ -181,6 +203,20 @@ function getBinaryProperties(featuresLength, properties, binaryBody) {
return binaryProperties;
}

function countBinaryPropertyMemory(binaryProperties) {
if (!defined(binaryProperties)) {
return 0;
}

let byteLength = 0;
for (const name in binaryProperties) {
if (binaryProperties.hasOwnProperty(name)) {
byteLength += binaryProperties[name].typedArray.byteLength;
}
}
return byteLength;
}

Cesium3DTileBatchTable.getBinaryProperties = function (
featuresLength,
batchTableJson,
Expand Down
4 changes: 3 additions & 1 deletion Source/Scene/Cesium3DTileContent.js
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,9 @@ Object.defineProperties(Cesium3DTileContent.prototype, {
},

/**
* Gets the amount of memory used by the batch table textures, in bytes.
* Gets the amount of memory used by the batch table textures and any binary
* metadata properties not accounted for in geometryByteLength or
* texturesByteLength
*
* @memberof Cesium3DTileContent.prototype
*
Expand Down
3 changes: 1 addition & 2 deletions Source/Scene/Cesium3DTileset.js
Original file line number Diff line number Diff line change
Expand Up @@ -1610,8 +1610,7 @@ Object.defineProperties(Cesium3DTileset.prototype, {

/**
* The total amount of GPU memory in bytes used by the tileset. This value is estimated from
* geometry, texture, and batch table textures of loaded tiles. For point clouds, this value also
* includes per-point metadata.
* geometry, texture, batch table textures, and binary metadata of loaded tiles.
*
* @memberof Cesium3DTileset.prototype
*
Expand Down
2 changes: 1 addition & 1 deletion Source/Scene/Cesium3DTilesetStatistics.js
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ function Cesium3DTilesetStatistics() {
// Memory statistics
this.geometryByteLength = 0;
this.texturesByteLength = 0;
this.batchTableByteLength = 0;
this.batchTableByteLength = 0; // batch textures and any binary metadata properties not otherwise accounted for
}

Cesium3DTilesetStatistics.prototype.clear = function () {
Expand Down
4 changes: 3 additions & 1 deletion Source/Scene/Geometry3DTileContent.js
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,9 @@ Object.defineProperties(Geometry3DTileContent.prototype, {

batchTableByteLength: {
get: function () {
return defined(this._batchTable) ? this._batchTable.memorySizeInBytes : 0;
return defined(this._batchTable)
? this._batchTable.batchTableByteLength
: 0;
},
},

Expand Down
2 changes: 1 addition & 1 deletion Source/Scene/GltfIndexBufferLoader.js
Original file line number Diff line number Diff line change
Expand Up @@ -217,7 +217,7 @@ GltfIndexBufferLoader.prototype.load = function () {
);
}

// Unload everything except the index buffer
// Unload everything except the index buffer and/or typed array.
loader.unload();

loader._buffer = buffer;
Expand Down
2 changes: 1 addition & 1 deletion Source/Scene/Instanced3DModel3DTileContent.js
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ Object.defineProperties(Instanced3DModel3DTileContent.prototype, {

ptrgags marked this conversation as resolved.
Show resolved Hide resolved
batchTableByteLength: {
get: function () {
return this._batchTable.memorySizeInBytes;
return this._batchTable.batchTableByteLength;
},
},

Expand Down
20 changes: 19 additions & 1 deletion Source/Scene/MetadataTable.js
Original file line number Diff line number Diff line change
Expand Up @@ -35,23 +35,27 @@ function MetadataTable(options) {
Check.typeOf.object("options.class", metadataClass);
//>>includeEnd('debug');

let byteLength = 0;
const properties = {};
if (defined(options.properties)) {
for (const propertyId in options.properties) {
if (options.properties.hasOwnProperty(propertyId)) {
properties[propertyId] = new MetadataTableProperty({
const property = new MetadataTableProperty({
count: count,
property: options.properties[propertyId],
classProperty: metadataClass.properties[propertyId],
bufferViews: options.bufferViews,
});
properties[propertyId] = property;
byteLength += property.byteLength;
}
}
}

this._count = count;
this._class = metadataClass;
this._properties = properties;
this._byteLength = byteLength;
}

Object.defineProperties(MetadataTable.prototype, {
Expand Down Expand Up @@ -82,6 +86,20 @@ Object.defineProperties(MetadataTable.prototype, {
return this._class;
},
},

/**
* The size of all typed arrays used in this table.
*
* @memberof MetadataTable.prototype
* @type {Number}
* @readonly
* @private
*/
byteLength: {
get: function () {
ptrgags marked this conversation as resolved.
Show resolved Hide resolved
return this._byteLength;
},
},
});

/**
Expand Down
Loading