Metadata: Add 3D metadata block to HDV once it's tested in demo and receives community feedback

[sbarbosadataverse]

Objective:

Provide support for 3D data metadata block.
Boston College collection on HDV is using a metadata block we added to their collection back in April 2022.

The fields in that metadata block have been previewed by Matt Cook's 3D data group project at Harvard Library.

Tasks:

• Determine if this block informed by Matt's group and associates will be used only in their collection or HDV wide, or if this block should be available as an external component or with the software
• Develop feedback process
• Matt Cook (@Cook4986) to add “IR Scanner” to scanning techniques
• Julian to create GitHub issue about adding 3D metadata block to Demo Dataverse, to be assigned to Steve Winship
• Julian and Sonia to meet about planning to test 3D metadata block

Resources:

Meeting Notes

Matt Cook's 3D schema and meeting notes with DV team:
https://docs.google.com/document/d/1_cZjth9x9lwyjRJfyG1KnHh6CamGqyRqmSEXFWlXlPk/edit
https://docs.google.com/spreadsheets/d/1XcLgys-j4G_GTKNT1choeW7FPc0l5x2IGY00aMw8Dtc/edit#gid=2017318209

3D Test Metadata Doc
https://docs.google.com/spreadsheets/d/1i1N4HnmmhXm27hNuenrehDiO8j2uLABW_lizhQl7Zlg/edit#gid=0

Example of 3D viewer in a Dataverse Installation (select a 3D file to view the 3D viewer)
https://dataverse.openforestdata.pl/dataset.xhtml?persistentId=doi:10.48370/OFD/Y8TNUO

Related Issue:

• Add 3D metadata block to Demo Dataverse for testing #319
• Add metadatablock for 3D data for Boston College #144

[cmbz]

2024/03/20

• Added to project, prioritized, and moved to Needs Sizing at request of @siacus

[sbarbosadataverse]

FYI I connected with Matt Cook on this issue. Waiting to hear back.

[jggautier]

Related to #144

[sbarbosadataverse]

see related issue to add 3D viewer to HDV: #256

[pdurbin]

Can we also add the 3D viewer to the list of external tools? https://guides.dataverse.org/en/latest/admin/external-tools.html#inventory-of-external-tools

This would involve creating an issue and pull request in the main repo, once we know the details of which 3D viewer we plan to use.

[Cook4986]

Some notes + links concerning 3D-data specific metadata (or "paradata"):

• Core "Capture" (i.e., production) and "File" (i.e., model) metadata fields, drawn from Harvard Library Imaging Services' 3D metadata gathering (Qulatrics) form and the file-level Morphosource metadata schema:
  • "3D Technique Used" or "Modality" (e.g., photogrammetry, RTI, structured light scanning, tomography, etc.)
  • "Equipment" or "Device(s)" (could include cameras, lenses, proprietary scanners, etc.)
  • "Poly Count" (i.e., detail/resolution)
  • "Viewer Link" (could link out to Sketchfab page, for example)
  • "Processing Software" (e.g., Metashape, RealityCapture, Polycam, etc.,)

[jggautier]

Hi all. If you haven't had the chance to yet, please review the guidelines at https://docs.google.com/document/d/1tY5t3gjrIgAGoRxVMWQSCh46fnbSmnFDLQ7aLkNLhJ8 for creating metadata blocks. Accounting for those guidelines early in the design process should help a lot. These guidelines are mentioned on the metadata customization page of Dataverse's Admin Guides. It's a living document of course so we welcome feedback about it.

And in case it's relevant here, when you're thinking about how to group similar fields together, consider keeping the number of any field's subfields to four or fewer. That way we can avoid some of the design issues that have come up, like the one described in #166. I'm considering adding this to the guidelines but haven't gotten around to it.

[Cook4986]

Thanks for sharing the very helpful guidelines doc link, @jggautier. I do think we can get this schema down to four subfields (by excluding the "Viewer Link" field, given the work being done with the X_ITE (pre)viewer (as per #56) and align the field titles with your text guidelines. What's next, to deploy a custom block in the HL 3D repo?

[jggautier]

Thanks @Cook4986.

I think @sbarbosadataverse will know the next steps.

[landreev]

OK, it's not yet clear if the block is ready to be installed in prod. in its current form in the repo. So, it will have to wait at least until next week. Moving it back into "This Sprint" for now.

[cmbz]

2024/07/10

• Checking in on readiness for this issue. Last updated in May 2024. @landreev @sbarbosadataverse

[landreev]

Please note that I am simply waiting to hear that the block is ready to be installed. Once ready, adding a block in production is trivial/can be done almost instantly.

But the version of the block we have in the repository (https://github.com/IQSS/dataverse.harvard.edu/blob/master/metadatablocks/custom3d.tsv) is 2 years old. And from a more recent discussion, above, it sounded like more changes were needed (?).
If you believe the block is ready to be installed as is, just let me know. But we got burned recently with a block that we put in production prematurely (it needed to be reorganized, but that became non-trivial because some of the fields had already been populated in the database), so it would make sense to confirm for sure that the users of the block are happy with it.

@sbarbosadataverse @jggautier @cmbz

[jggautier]

That TSV file, https://github.com/IQSS/dataverse.harvard.edu/blob/master/metadatablocks/custom3d.tsv, is not the metadata block being referred to in this GitHub issue, although it was added to help folks from Boston University describe 3D data they'll publish in Harvard Dataverse, so I mentioned in March that I think it's related to the work being tracked in this GitHub issue.

I don't have much information about the progress of the 3D metadata block we're referring to in this GitHub issue. I see a spreadsheet that @sbarbosadataverse added in a comment. And notes that Matt Cook wrote in April and in May.

[landreev]

Ok, so it sounds like we don't have a block to install yet.
You guys may want to consider changing the title to "develop a 3D metadata block for HDV". It's still not clear to me who, or, if anyone is working on creating this block - but maybe re-scheduling it formally as such may help (?).
My only involvement with this issue was under a mistaken assumption that it was a devops one, for activating an already existing block in prod. I'm pretty positive that that was the reason it was sized as a "3". It's obviously more work than that, if this is about creating a new block.

@sbarbosadataverse @jggautier @cmbz

[sbarbosadataverse]

I pinged Matt Cook today to schedule a meeting to review the block for any updates.

[cmbz]

2024/07/17

• Moved to On Hold, pending response from Matt

[sbarbosadataverse]

A meeting with Matt is scheduled for next week July 25th , and we have additional metadata to consider for the Block:

"3D Technique Used" or "Modality" (e.g., photogrammetry, RTI, structured light scanning, tomography, etc.)
Equipment" or "Device(s)" (could include cameras, lenses, proprietary scanners, etc.)
"Poly Count" (i.e., detail/resolution)
"Processing Software" (e.g., Metashape, RealityCapture, Polycam, etc.,)

[jggautier]

I'm just adding an update that the meeting was rescheduled. It's today, Aug 15 at 4pm.

[jggautier]

Matt (@Cook4986) and I met on Aug 15. Meeting notes are at https://docs.google.com/document/d/1_cZjth9x9lwyjRJfyG1KnHh6CamGqyRqmSEXFWlXlPk.

We talked about others who've worked on describing 3D data and I said I'd provide contact info. It's in the meeting notes. Matt said he'll get back to us when he's ready to share a metadata block he's working on, looking at what's been created for the Boston College collection.

[jggautier]

We're waiting to hear back from Matt when he's ready to share a metadata block he's working on, looking at what's been created for the Boston College collection.

[sbarbosadataverse]

Can we also add the 3D viewer to the list of external tools? https://guides.dataverse.org/en/latest/admin/external-tools.html#inventory-of-external-tools

This would involve creating an issue and pull request in the main repo, once we know the details of which 3D viewer we plan to use.

Just making note this is complete

[jggautier]

Hi @Cook4986. Could I help add the metadata block TSV file to this GitHub repository? I think this might make it easier to share it and track changes to it. And If we decide to make it available to other Dataverse installations, too, we could always move it to the main Dataverse repository later on.

[jggautier]

Over email @Cook4986 agreed that I could turn the Excel spreadsheet we've been working on into a TSV file, make the remaining change (adding “IR Scanner” to the list of scanning techniques), and add it to GitHub to continue working on it and make it easier to add to Demo Dataverse to help with learning how well the new fields work.

@sbarbosadataverse let me know that @siacus asked that we add the metadata block to the metadata blocks that ship with Dataverse, as opposed to making it a metadata block that's exclusive to collections in Harvard Dataverse.

@sbarbosadataverse and @cmbz, I'm revising the testing plan I've proposed to prioritize implementation, that is adding the metadata block to the metadata blocks that ship with Dataverse. The testing and more passive forms of feedback collection (such as asking in the Google Groups forum for feedback) can happen as soon as we have ways for others to experience the new fields (like adding the metadata block to Demo Dataverse (#319)). And testing and more feedback collection won't need to happen before the metadata block is added to the other metadata blocks that ship with Dataverse.

I think that this will increase the chances that the metadata block can be made available sooner for @Cook4986 and his colleagues at Harvard Library to use in their collection in Harvard Dataverse, maybe even included in the version of Dataverse that comes right after v6.5.

If you agree with this, I think we should close this issue, since the metadata block won't be exclusive to collections in Harvard Dataverse and we won't be waiting for testing and more feedback before adding the metadata block. Once the metadata block ships with Dataverse and Harvard Dataverse is updated to the Dataverse release that includes that metadata block, all collections in Harvard Dataverse can use the new metadata block.

And I can open the following GitHub issues:

• a GitHub issue in the Dataverse GitHub repo about adding the metadata block to the metadata blocks that ship with Dataverse
• a GitHub issue about planning for how to replace the "custom3d" metadata block being used by Boston College's collection, with the updated metadata block that will be available for all Dataverse users

[cmbz]

@jggautier I'm on board with your plan described here: #253 (comment)

To make sure the issues don't get lost, please add them to Needs Sizing and add to the Sprint Planning document so they are discussed in the Jan 2 sprint planning meeting.

[jggautier]

Okay, I created the GitHub issues (IQSS/dataverse#11120, #329) and added them to the Needs Sizing column on the Sprint Planning board.