Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't map back to Julia composite type #504

Closed
koehlerson opened this issue Nov 15, 2023 · 2 comments
Closed

Don't map back to Julia composite type #504

koehlerson opened this issue Nov 15, 2023 · 2 comments

Comments

@koehlerson
Copy link
Contributor

koehlerson commented Nov 15, 2023

Let's say I have a struct called Foo and I save one instance to disk

julia> struct Foo
           x::Int
           z::Float64
       end

julia> jldopen("blah.jld2","w") do jld
           jld["test"] = Foo(1,2.0)
       end
Foo(1, 2.0)

if I close the session, open a new one and slightly redefine the type by only adding a parametrization (even though the data needed to reconstruct is still there) I cannot read the data in the form of x and z values

julia> using JLD2

julia> struct Foo{T1,T2}
           x::T1
           z::T2
       end

julia> file = jldopen("blah.jld2","r")
JLDFile /home/mkoehler/blah.jld2 (read-only)
 └─🔢 test

julia> file["test"]
┌ Warning: read type Foo is not a leaf type in workspace; reconstructing
└ @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/data/reconstructing_datatypes.jl:306
Reconstruct@Foo((1, 2.0))

Is there a way such that I don't remap the stored data to Foo but instead just read the plain data, i.e. x and z in this case?
Basically, what I want is HDF5.jl behavior as a possible option:

julia> h5open("blah.h5","w") do hdf5
           hdf5["test"] = [Foo(1,2.0)]
       end
1-element Vector{Foo{Int64, Float64}}:
 Foo{Int64, Float64}(1, 2.0)

julia> h5open("blah.h5","r") do hdf5
           hdf5["test"][]
       end
1-element Vector{NamedTuple{(:x, :z), Tuple{Int64, Float64}}}:
 (x = 1, z = 2.0)

If I open the file with HDF5 it works as I intend:

julia> h5open("blah.jld2","r") do jld2
           jld2["test"][]
       end
(x = 1, z = 2.0)

but I think it would be nice to have the optional type remapping when reading a file with JLD2. I know that an explicit type remapping is possible as well as doing a custom serialization, but as far as I know the custom serialization option always remaps to a composite type back and I have some use cases where I don't want this all the time, but instead I'm okay to look at (x=1,z=2.0).

@JonasIsensee
Copy link
Collaborator

Hi @koehlerson,

I will admit, that this is not obvious but with
https://juliaio.github.io/JLD2.jl/dev/advanced/

you can do

julia> load("blah.jld2"; typemap=Dict("Main.Foo"=>JLD2.Upgrade(NamedTuple)))
Dict{String, Any} with 1 entry:
  "test" => (x = 1, z = 2.0)

@koehlerson
Copy link
Contributor Author

Thanks for the hint!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants