None types are not preserved

[drdavella]

As @vmarkovtsev pointed out in #534, None types are currently stripped from the tree.

I don't think the standard has anything to say about this at the moment. Would it be useful if None values are preserved in the tree? This would probably require the definition of a None or Null tag. I need to think through the implications a little more.

cc @perrygreenfield

[drdavella]

This appears to be an issue with ASDF and not with the underlying YAML implementation:

>>> import yaml
>>> tree = dict(a=1, b='hello', c=None)
>>> result = yaml.dump(tree)
>>> result
'a: 1\nb: hello\nc: null\n'
>>> yaml.safe_load(result)
{'a': 1, 'b': 'hello', 'c': None}

So this is probably a bug and should be addressed.

[StijnDebackere]

I just ran into this issue as well. I think things go wrong when converting the tree to a tagged tree.

>>> import asdf
>>> tree = {"a": 1, "b": None}
>>> af = asdf.AsdfFile(tree)
>>> af.tree
{'a': 1}
>>> tree_tagged = asdf.yamlutil.custom_tree_to_tagged_tree(tree, af)
>>> tree_tagged
{'a': 1}
>>> tree
{'a': 1, 'b': None}

I haven't been able to check, but it seems like in line 138 of treeutil.walk_and_modify there is an explicit check that does not allow dict values to be None. I don't know whether removing this check would cause any other issues that I cannot oversee?

[eslavich]

The callback is allowed to return None to indicate that the key should be removed. It seems like that behavior is intentional, since there's a test for it.

@jdavies-st maybe we should define a special value that can be returned instead of None, something analogous to NotImplemented?

[jdavies-st]

It's not clear why None is being treated as a special case. Certainly None is used in special ways in python, but it is not clear that asdf should be treating it specially.

The fact that serialization with None values does not round trip is definitely not desirable.

[jdavies-st]

So based on discussions with Perry and an email from Mike, it seems the driver of None being dropped is that it is interpreted as a "missing" value. This is driven by the fact that ASDF allows the definition of a default value in the schema. So if the attribute in the tree is missing, but the schema provides a default value, internally None signifies it is missing, and it gets filled with the default value.

If you're not using schemas to validate your ASDF tree, then this probably unexpected.

I think we need to think more about how we might make None round-trip-able and still indicate a value is "missing" and therefore might need a default value that the schema provides.

[aprsa]

This just bit me as I expected None to be stored in the tree but it wasn't and my parser choked on a non-existing key. Would it not be better to perhaps tag missing values with nan and leave None intact?

[perrygreenfield]

I think some solution is needed to deal with this. NaN is probably not the right solution but perhaps some asdf-defined special value indicating missing.

[aprsa]

Alternatively, would you consider implementing a get() method (similar to dict's get()), so that we could do:

data = asdf.open('file.asdf')
var = data.get('key', None)