Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Definition of axis ... in a NXdata #755

Closed
2 tasks done
kif opened this issue Feb 17, 2020 · 16 comments
Closed
2 tasks done

Definition of axis ... in a NXdata #755

kif opened this issue Feb 17, 2020 · 16 comments
Labels
Milestone

Comments

@kif
Copy link

kif commented Feb 17, 2020

Hi there,

If I understand well, the NXdata is intended to describe the plot (i.e. what to plot, how, ...). I need to save some SAXS data so that they can be displayed by default correctly, following the Nexus convention.
In such plot, the intensity I should be plotted in 1D vs the scattering vector q:

  • How to indicate to plot I (and sometimes q) in log-scale ? Apparently Nexus does not address this use-case, does it ?
  • How to indicate to the plot that the intensity is in inverse steradian or in inverse cm. The same is true for q which is likely to be inverse nm or inverse angstrom ?

I found in the Nexus documentation that the x dataset in an NXdata can have an attribute @Units but it looks to be part of an obsolete definition. Would you consider this saxs NXdata as valid from a Nexus perspective ?

- saxs
  @NX_class = NXdata
  @axes = ["q"]
  @signal = "I"
  @interpretation = "spectrum"
  
  I: dataset float[1000]
   @units = "1/sr"

  q: dataset float[1000]
    @units = "1/A"
  

Some references:
silx-kit/silx#2927

@prjemian
Copy link
Contributor

How to indicate to plot I (and sometimes q) in log-scale ? Apparently Nexus does not address this use-case, does it ?

That's correct. NeXus has decided to not define whether (or not) to use log-scale.

@prjemian
Copy link
Contributor

How to indicate to the plot that the intensity is in inverse steradian or in inverse cm. The same is true for q which is likely to be inverse nm or inverse angstrom ?

Your example is correct. Use the @units attribute.

There is one thing to change with your example. It is common practice in small-angle scattering to use @units="1/A" for Q. Yet this looks like inverse electrical current (Amperes). As the NeXus documentation states:

At present, the contents of NeXus units attributes are not validated in data files.

The NXcanSAS definition has arrived at the use of 1/angstrom as the preferred spelling. This spelling is recognized by the Unidata UDUNITS-2 library. Use of the preferred spelling will aid data processing software that implements recognition and conversion of units.

@prjemian
Copy link
Contributor

The @interpretation attribute is not reserved by NeXus (except in NXcxi_ptycho which usage is slightly different than here). Thus, this is a good example of a custom attribute. Keep in mind that custom attributes might not be recognized or used by generalized software that reads NeXus data files.

@vasole
Copy link
Contributor

vasole commented Feb 17, 2020

The @interpretation attribute is reserved by NeXus for datasets. It is the first time I see it associated to an NXdata group.

@prjemian
Copy link
Contributor

I found in the Nexus documentation that the x dataset in an NXdata can have an attribute @Units but it looks to be part of an obsolete definition.

Do you recall why this looked obsolete?

The engineering @units attribute is standard NeXus. It can be applied to any NeXus field. In a NeXus NXDL specification, fields with engineering units have declarations of what category of units should be provided, appropriate to the field. The list of categories is given in the manual section: Unit Categories allowed in NXDL specifications. For example, the Q units category used by NXcanSAS is: NX_PER_LENGTH.

Your questions here prompted me to use the index of the NeXus documentation to find units. In the index, the reference highlighted with bold style (https://manual.nexusformat.org/datarules.html#index-19) indicates the principal reference.

@prjemian
Copy link
Contributor

The @interpretation attribute is reserved by NeXus for datasets. It is the first time I see it associated to an NXdata group.

Where is this written in the manual?

@prjemian
Copy link
Contributor

And, technically, monochromatic SAXS data is not a spectrum.

@prjemian
Copy link
Contributor

Ok, I see it now. That's definitive. And, spectrum is the string defined by NeXus to describe 1-D data. This is consistent with a common definition of the word. My understanding of the word spectrum is more narrow, requiring a range of wavelengths (such as definition 1e from above).

Here are the references to interpretation in the NeXus manual:

@prjemian
Copy link
Contributor

As described in the manual, @interpretation is an attribute of a field. While it is not defined for use in a group, it should not be used with a group because it is already reserved for other use.

@vasole
Copy link
Contributor

vasole commented Feb 17, 2020

Yes Pete, it has nothing to do with what the data actually represent. It is just a convention for specifying 1D data. I recall that Mark preferred the word histogram, but finally spectrum was retained.

@prjemian
Copy link
Contributor

@kif - feel free to close this issue if all your questions have been answered.

@kif
Copy link
Author

kif commented Feb 17, 2020 via email

@prjemian
Copy link
Contributor

If you used the unicode Å (\u212B), you might find interchange with other software might be more difficult unless they expect this. Times are changing and unicode is more widely accepted. For me, the test is Will it be convertible using the cfunits package? That package implements UDUNITS in python.

setup the conda python environment

(base) mintadmin@mint-vm:~/.../sandbox/apstools$ conda create -y -n units python ipython cfunits -c conda-forge
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/mintadmin/Apps/anaconda/envs/units

  added / updated specs:
    - cfunits
    - ipython
    - python


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    certifi-2019.11.28         |           py38_0         148 KB  conda-forge
    cftime-1.0.4.2             |   py38hc1659b7_0         327 KB  conda-forge
    ipython-7.12.0             |   py38h5ca1d4c_0         1.1 MB  conda-forge
    jedi-0.16.0                |           py38_0         782 KB  conda-forge
    ld_impl_linux-64-2.33.1    |       h53a641e_8         589 KB  conda-forge
    netcdf4-1.5.3              |nompi_py38hd35fb8e_102         580 KB  conda-forge
    numpy-1.18.1               |   py38h95a1406_0         5.3 MB  conda-forge
    pexpect-4.8.0              |           py38_0          79 KB  conda-forge
    pickleshare-0.7.5          |        py38_1000          12 KB  conda-forge
    setuptools-45.2.0          |           py38_0         655 KB  conda-forge
    six-1.14.0                 |           py38_0          23 KB  conda-forge
    traitlets-4.3.3            |           py38_0         134 KB  conda-forge
    ------------------------------------------------------------
                                           Total:         9.6 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-0_gnu
  backcall           conda-forge/noarch::backcall-0.1.0-py_0
  bzip2              conda-forge/linux-64::bzip2-1.0.8-h516909a_2
  ca-certificates    conda-forge/linux-64::ca-certificates-2019.11.28-hecc5488_0
  certifi            conda-forge/linux-64::certifi-2019.11.28-py38_0
  cftime             conda-forge/linux-64::cftime-1.0.4.2-py38hc1659b7_0
  cfunits            conda-forge/noarch::cfunits-3.1.1-py_0
  curl               conda-forge/linux-64::curl-7.68.0-hf8cf82a_0
  decorator          conda-forge/noarch::decorator-4.4.1-py_0
  expat              conda-forge/linux-64::expat-2.2.9-he1b5a44_2
  hdf4               conda-forge/linux-64::hdf4-4.2.13-hf30be14_1003
  hdf5               conda-forge/linux-64::hdf5-1.10.5-nompi_h3c11f04_1104
  ipython            conda-forge/linux-64::ipython-7.12.0-py38h5ca1d4c_0
  ipython_genutils   conda-forge/noarch::ipython_genutils-0.2.0-py_1
  jedi               conda-forge/linux-64::jedi-0.16.0-py38_0
  jpeg               conda-forge/linux-64::jpeg-9c-h14c3975_1001
  krb5               conda-forge/linux-64::krb5-1.16.4-h2fd8d38_0
  ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.33.1-h53a641e_8
  libblas            conda-forge/linux-64::libblas-3.8.0-14_openblas
  libcblas           conda-forge/linux-64::libcblas-3.8.0-14_openblas
  libcurl            conda-forge/linux-64::libcurl-7.68.0-hda55be3_0
  libedit            conda-forge/linux-64::libedit-3.1.20170329-hf8c457e_1001
  libffi             conda-forge/linux-64::libffi-3.2.1-he1b5a44_1006
  libgcc-ng          conda-forge/linux-64::libgcc-ng-9.2.0-h24d8f2e_2
  libgfortran-ng     conda-forge/linux-64::libgfortran-ng-7.3.0-hdf63c60_5
  libgomp            conda-forge/linux-64::libgomp-9.2.0-h24d8f2e_2
  liblapack          conda-forge/linux-64::liblapack-3.8.0-14_openblas
  libnetcdf          conda-forge/linux-64::libnetcdf-4.7.3-nompi_h9f9fd6a_101
  libopenblas        conda-forge/linux-64::libopenblas-0.3.7-h5ec1e0e_6
  libssh2            conda-forge/linux-64::libssh2-1.8.2-h22169c7_2
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-9.2.0-hdf63c60_2
  ncurses            conda-forge/linux-64::ncurses-6.1-hf484d3e_1002
  netcdf4            conda-forge/linux-64::netcdf4-1.5.3-nompi_py38hd35fb8e_102
  numpy              conda-forge/linux-64::numpy-1.18.1-py38h95a1406_0
  openssl            conda-forge/linux-64::openssl-1.1.1d-h516909a_0
  parso              conda-forge/noarch::parso-0.6.1-py_0
  pexpect            conda-forge/linux-64::pexpect-4.8.0-py38_0
  pickleshare        conda-forge/linux-64::pickleshare-0.7.5-py38_1000
  pip                conda-forge/noarch::pip-20.0.2-py_2
  prompt_toolkit     conda-forge/noarch::prompt_toolkit-3.0.3-py_0
  ptyprocess         conda-forge/noarch::ptyprocess-0.6.0-py_1001
  pygments           conda-forge/noarch::pygments-2.5.2-py_0
  python             conda-forge/linux-64::python-3.8.1-h357f687_2
  readline           conda-forge/linux-64::readline-8.0-hf8c457e_0
  setuptools         conda-forge/linux-64::setuptools-45.2.0-py38_0
  six                conda-forge/linux-64::six-1.14.0-py38_0
  sqlite             conda-forge/linux-64::sqlite-3.30.1-hcee41ef_0
  tk                 conda-forge/linux-64::tk-8.6.10-hed695b0_0
  traitlets          conda-forge/linux-64::traitlets-4.3.3-py38_0
  udunits2           conda-forge/linux-64::udunits2-2.2.27.6-h4e0c4b3_1001
  wcwidth            conda-forge/noarch::wcwidth-0.1.8-py_0
  wheel              conda-forge/noarch::wheel-0.34.2-py_1
  xz                 conda-forge/linux-64::xz-5.2.4-h14c3975_1001
  zlib               conda-forge/linux-64::zlib-1.2.11-h516909a_1006



Downloading and Extracting Packages
ipython-7.12.0       | 1.1 MB    | ################################################################################################# | 100% 
traitlets-4.3.3      | 134 KB    | ################################################################################################# | 100% 
ld_impl_linux-64-2.3 | 589 KB    | ################################################################################################# | 100% 
setuptools-45.2.0    | 655 KB    | ################################################################################################# | 100% 
pexpect-4.8.0        | 79 KB     | ################################################################################################# | 100% 
six-1.14.0           | 23 KB     | ################################################################################################# | 100% 
netcdf4-1.5.3        | 580 KB    | ################################################################################################# | 100% 
pickleshare-0.7.5    | 12 KB     | ################################################################################################# | 100% 
cftime-1.0.4.2       | 327 KB    | ################################################################################################# | 100% 
numpy-1.18.1         | 5.3 MB    | ################################################################################################# | 100% 
jedi-0.16.0          | 782 KB    | ################################################################################################# | 100% 
certifi-2019.11.28   | 148 KB    | ################################################################################################# | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate units
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Test various units

(base) mintadmin@mint-vm:~/.../sandbox/apstools$ conda activate units
(units) mintadmin@mint-vm:~/.../sandbox/apstools$ ipython
Python 3.8.1 | packaged by conda-forge | (default, Jan 29 2020, 14:55:04) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.12.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from cfunits import *                                                                                                               

In [2]: Units("feet")                                                                                                                       
Out[2]: <Units: feet>

In [3]: 1/(1/Units("feet"))                                                                                                                 
Out[3]: <Units: 0.3048 m>

In [4]: 1/Units("feet")                                                                                                                     
Out[4]: <Units: 3.28083989501312 m-1>

In [5]: 1/(1/Units("Å"))                                                                                                                    
Out[5]: <Units: 1e-10 m>

In [6]: 1/(1/Units("\u212B"))                                                                                                               
Out[6]: <Units: 1e-10 m>

In [7]: 1/(1/Units("angstrom"))                                                                                                             
Out[7]: <Units: 1e-10 m>

In [8]: 1/Units("Å") == 1/Units("angstrom")                                                                                                 
Out[8]: True

In [9]: 1/Units("\u212B") == 1/Units("angstrom")                                                                                            
Out[9]: True

In [10]: 1/Units("nm") == 1/Units("angstrom")                                                                                               
Out[10]: False

Since NeXus does not validate the engineering @units attribute now, and cfunits recognizes "Å", "\u212B", and "angstrom" as equal, what reasonable objection can be raised? I would accept just as you have written.

@rayosborn
Copy link
Contributor

rayosborn commented Feb 18, 2020

I may have misunderstood what is being suggested, but you can't store unicode directly in HDF5 files. However, you can store unicode strings as variable-length strings encoded as UTF-8 (unless a 'byte' dtype is specifically chosen, in which case I guess you can choose your own encoding - it's unwise to use other encodings since there is no way of specifying which you are using). Programs like h5py will automatically decode variable-length strings into unicode strings. These days, I don't think there is any reason NeXus fields and attributes shouldn't contain unicode strings although the documentation doesn't mention them. Å makes sense to me.

@vasole
Copy link
Contributor

vasole commented Feb 18, 2020

I think @rayosborn has made a point that was clearly stated in NIAC 2018 and that made it into the documentation. Therefore the UTF-8 representation of the Angstrom unit is perfectly valid and in agreement with the NeXus manual:

NX_CHAR

The preferred string representation is UTF-8. Both fixed-length strings and variable-length strings are valid. String arrays cannot be used where only a string is expected (title, start_time, end_time, NX_class attribute,…). Fields or attributes requiring the use of string arrays will be clearly marked as such (like the NXdata attribute auxiliary_signals).

@kif
Copy link
Author

kif commented Feb 20, 2020

Thanks for the comments

@kif kif closed this as completed Feb 20, 2020
@prjemian prjemian added this to the NXDL 2020.10 milestone Feb 25, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants