0.6.0 Release Target

[jreadey]

This issue is to document the set of features/bug fixes for the next (v0.6) HDF Compass Release

• Tree View for object hierarchy - Display Object hierarchy in Tree #66
• Fix OpenDAP plugin: OPeNDAP plug-in issues #60
• Recent file history - PR: Added Open Recent file menu #50
• User Guide - Provide User Guide #76
• Better support for attribute display - Better support for attribute display #72
• Better support for viewing HDF5 images - Better support for viewing HDF5 Images  #23
• Provide display of available Plugins - Provide display of Plugins available #73
• External/Soft links support - External/Soft links don't display as links #77
• Clipboard Support - Clipboard support #21
• Provide h5serv resource plugin - Provide plugin for HDF Server resources #78
• Improve display of linked datatypes - Improve the display of linked datatypes #25
• Show attribute count - Show attribute count in Group/Dataset/Datatype overview #74
• Provide Text/XML view - Extension of compass_model for text and xml #69
• Improve display of committed types - Display info for Committed Types #75
• Python 2-3 Compatibility - Python 3 compatibility #46
• Provide logging support - PR adoption of Python logging mechanism #64
• Disable plot for string types - Disable Plot for string types #70
• Fix display of string element - Only first line of string elements displayed #44
• Follow PEP8 guidelines: PEP8 coding style #52
• BAG plugin: BAG plugin #80

See issue #61 for discussion of many of these items.

Excluded from this list (but intended for the next release) are some items (e.g. code signing for Mac binaries) that are a HDFGroup build/deploy process and not a source code issue per se.

[ghost]

@jreadey You can add checkbox for each of the issues and track progress (how many are checked vs. total). See the first listed issue.

[jreadey]

Builds off the latest dev branch checkins are available here:

• Windows - https://s3.amazonaws.com/hdfgroup.download/HDFCompass/latest/hdfcompass-latest-win_amd64.exe
• Mac OS X - https://s3.amazonaws.com/hdfgroup.download/HDFCompass/latest/hdfcompass-latest-macosx.dmg
• Linux - https://s3.amazonaws.com/hdfgroup.download/HDFCompass/latest/hdfcompass-latest-centos_amd64

Please submit any problems you find as an issue (or update an existing issue). If you can run from the source tree, it will be helpful to note if this problem only shows up in install executable or is present in the source version as well.

[jreadey]

Builds for Windows and Mac are updated:

Windows - https://s3.amazonaws.com/hdfgroup.download/HDFCompass/latest/hdfcompass-latest-win_amd64.exe
Mac OS X - https://s3.amazonaws.com/hdfgroup.download/HDFCompass/latest/hdfcompass-latest-macosx.dmg

BAG plugin in (based on my limited testing) is working with this build.
I ran into an issue building on Linux - I'll get a new Linux build out later in the week.

[hyoklee]

Mac binary works well even for https://eosdap OPeNDAP server. How did you fix it?

[giumas]

@jreadey How did you freeze on Mac? Py2app or PyInstaller? I am asking in relation to #107

[giumas]

Although not listed in the milestones, the following PRs add/fix the unittest for the current models:

• Fix test_icons + array_model tests and __init__.py #110
• Add tests and fix __init__.py for asc_model #111
• Add tests to bag_model + fix __init__ for hdf5_model #112
• Fix __init__.py for filesystem_model and opendap_model (+tests) #113

Some additional work could be done to implement tests for other structures (since only Store and Container are currently implemented) or to add tests for the packages that are not models (e.g. hdf_compass.utils and hdf_compass.compass_viewer). However, the current situation could be good enough for the next release.

[jreadey]

@giumas - I used PyInstaller to freeze the binary on Mac. With py2app I was hitting the problem you described in #107. Using PyInstaller introducing the problem that @andrewcollette mentioned - dragging files to the toolbar icon does not work. I've created an issue for this: #116.

@hyoklee - I didn't do anything special to fix the OpenDAP problem. Maybe it was a beneficial effect of the switch to PyInstaller?

[hyoklee]

@jreadey Thanks for explaining how you packaged. I noticed that Mac's OPeNDAP problem is related to openssl used in requests module under Pydap. If requests is built with openssl 1.0.2.d, Pydap works OK with eosdap server. If it is built with 0.9.8zg, Pydap returns SSL error.

[giumas]

@jreadey Here there are more details on this issue: pyinstaller/pyinstaller#1309

So we could monitor both issues (the PyInstaller and the Py2App) to see what will be fixed first.

[jreadey]

Builds for Windows and Mac are updated:

Windows - https://s3.amazonaws.com/hdfgroup.download/HDFCompass/latest/hdfcompass-latest-win_amd64.exe
Mac OS X - https://s3.amazonaws.com/hdfgroup.download/HDFCompass/latest/hdfcompass-latest-macosx.dmg

[giumas]

@jreadey I have just added some minimal changes to make optional (at runtime) the cartopy dependency (due to troubles to freeze it on OS X). Here I have uploaded frozen binaries for Mavericks, Yosemite, and Windows: https://github.com/giumas/hdf-compass/releases/tag/0.6.0-beta.1
Let me know if they work!

[giumas]

@jreadey did you have time to look at the binaries?

[jreadey]

@giumas - sorry I haven't had time to work on Compass recently.

I did try freezing the current dev branch on Windows 7 today. I get a pyinstaller error though, "ImportError: no module named cartopy". So I guess that pyinstaller is building the list of imports by scanning the source files, and thus picks up the cartopy anyway.

Do you have access to a Windows 7 box without Cartopy to try this on?

Also, are you able to freeze Compass on OS X?

[giumas]

It is a kind of expected behavior.. You first install cartopy and all the other dependencies and freeze the application, then if there are issues at runtime in loading the cartopy library, HDF Compass does not crash in displaying the geo-arrays, but reverts to plain matplotlib.

I did not have issues to freeze on Win7, the issue with cartopy was on OS X. Specifically, there were two different versions of geos in use by the dependencies, and pyinstaller seems to pick the oldest one. I did not have times to look more on it.. It may be that you with a different environment can get the freezing smoothly.

However, did you try these frozen applications? https://github.com/giumas/hdf-compass/releases/tag/0.6.0-beta.1

[giumas]

This is the trick that I have added for cartopy: https://github.com/giumas/hdf-compass/blob/b951e784b97da81660fdbd68b6f894c002d05e5a/hdf_compass/bag_model/model.py#L389

[jreadey]

I tried the frozen Yosemite application and get the following error. Have you tried it on any Mac that didn't have cartopy installed?

/Applications/HDFCompass.app/Contents/MacOS/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
Trying `CDLL(/Applications/HDFCompass.app/Contents/MacOS/HDFCompass/libgeos_c.dylib)`
DEBUG   shapely.geos.load_dll:46 > Trying `CDLL(/Applications/HDFCompass.app/Contents/MacOS/HDFCompass/libgeos_c.dylib)`
Failed `CDLL(/Applications/HDFCompass.app/Contents/MacOS/HDFCompass/libgeos_c.dylib)`
WARNING shapely.geos.load_dll:50 > Failed `CDLL(/Applications/HDFCompass.app/Contents/MacOS/HDFCompass/libgeos_c.dylib)`
Traceback (most recent call last):
  File "<string>", line 36, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "hdf_compass/compass_viewer/__init__.py", line 14, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "hdf_compass/compass_viewer/viewer.py", line 34, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "hdf_compass/compass_viewer/geo_surface/__init__.py", line 14, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "hdf_compass/compass_viewer/geo_surface/frame.py", line 27, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "hdf_compass/compass_viewer/geo_surface/plot.py", line 21, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "cartopy/__init__.py", line 24, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "shapely/speedups/__init__.py", line 3, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "shapely/geometry/__init__.py", line 4, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "shapely/geometry/base.py", line 9, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "shapely/coords.py", line 8, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "shapely/geos.py", line 124, in <module>
  File "shapely/geos.py", line 61, in load_dll
OSError: Could not find lib geos_c or load any of its variants ['/Applications/HDFCompass.app/Contents/MacOS/HDFCompass/libgeos_c.dylib'].
HDFCompass returned -1

[giumas]

Mmm.. I guess that I set the wrong exception: https://github.com/HDFGroup/hdf-compass/blob/develop/hdf_compass/compass_viewer/geo_surface/plot.py#L23

It should be OSError in place of ImportError (or a wider Exception). And you are surely right that it was finding the geos library in somehow on my machine.

However, the straight solution would be to fix the geos issue.. We may try with opening a pyinstaller ticket. I will try to put together a minimum example with such an issue.

@QuLogic: I saw that you are an active contributor of cartopy. Have you never successfully frozen an application with cartopy on OS X?

[QuLogic]

No, I don't use OS X. This geos library thing seems to come up a lot there; maybe ask @pelson.

[pelson]

I'm afraid I don't have any experience of freezing applications either. Looking at the traceback, it looks suspiciously like a shapely issue I have seen go by... shapely/shapely#368

Please feel free to ping me if there is anything I can do.

Thanks,

[giumas]

@pelson and @QuLogic : I have finally been able to solve the issue with freezing HDFCompass on OS X with the new cartopy and shapely dependencies by:

• using the development branch and building shapely with the HomeBrew's geos (in place of the wheel) as discussed in many tickets: cascaded_union stopped working on LineString shapely/shapely#273, OS X 10.10 cartopy installation produces segmentation fault when calling shapely-geos SciTools/cartopy#615, OS X 10.10 cartopy installation produces segmentation fault when calling shapely-geos SciTools/cartopy#615
• adding an additional fallback path for geos as you can read in this PR: PyInstaller's OS X-specific bug fix shapely/shapely#382

About this last PR, I am not sure if it would be the case to delete the existing pyinstaller path: https://github.com/Toblerity/Shapely/blob/master/shapely/geos.py#L88 since it might be never valid.
However, I am not confident on doing this deletion, so I am pinging @tallforasmurf and @htgoebel on the subject. I believe that is in somehow related to pyinstaller/pyinstaller#1792 and fixing this part on the shapely side would avoid to have to do it in the shapely hook.

[jreadey]

Preview 0.6 binaries for OS X (Mountain Lion or later) and Windows 7 (or later) are available here:

Windows: https://s3.amazonaws.com/hdfgroup.download/HDFCompass/0.6.0b5/HDFCompass.exe

OS X: https://s3.amazonaws.com/hdfgroup.download/HDFCompass/0.6.0b5/HDFCompass.dmg

Please try these out and comment here if you find problems. If there are no show stoppers this will go out as the 0.6 release.

[oyarzun]

OS X file seems to be corrupt.

[jreadey]

@oyazun, sorry about that. Try it now.

[oyarzun]

@jreadey Thanks that one works.

[jreadey]

A new preview binary is available that includes @giumas' fix for the OpenDAP plugin:

Windows: http://hdfgroup.download.s3.amazonaws.com/HDFCompass/0.6.0b6/HDFCompass.exe
OS X: http://hdfgroup.download.s3.amazonaws.com/HDFCompass/0.6.0b6/HDFCompass.dmg

Everyone, please post here if you see any problems.

[giumas]

It works on Win7 x64

[jreadey]

Did you see have any problems with the url's mentioned in #60?

[giumas]

https://eosdap.hdfgroup.org:8080/opendap/data/hdf5/grid_1_2d.h5 works
http://test.opendap.org/dap/data/nc/coads_climatology.nc works

however, the issue that I wrote on #60 is still there:

I have looked a bit on it.. and I suspect that the issue is due to the fact that we are using wxFileHistory to store urls: https://github.com/HDFGroup/hdf-compass/blob/develop/hdf_compass/compass_viewer/frame.py#L87. This would agree with the fact that I have this issue only with urls.

A dirty fix could be to only have the history for files.. A better way would be to create a FileUrlHistory class, but I don't have currently time to look at it. @jreadey , do you remember who added this functionality? he/she may have more insides

[giumas]

I have attempted to comment out this row, and then I am not more getting the error. So it should be that!

[ghost]

@jreadey

Everyone, please post here if you see any problems.

My Mac says: "HDFCompass.app" is damaged or corrupted" and cannot be opened. Tried three times with downloading the dmg file.

[jreadey]

@ajelenak-thg - I just tried again downloading from this link: http://hdfgroup.download.s3.amazonaws.com/HDFCompass/0.6.0b6/HDFCompass.dmg. I got the security warning because it is unassigned but otherwise no problems.

Has anyone else tried the OS X version above?

[giumas]

It works for me on OS X Mavericks Server:

[giumas]

As well as on OS X Yosemite:

[giumas]

@jreadey Incidentally, I noticed that there are some minor inconsistencies in the naming.. like the link sayes 0.6.b6, the about 0.6.b5, the mounted dmg 0.6.0b3, etc.

[ghost]

Nope, does not work for me. Still the same "may be damaged or incomplete" error. OS X Yosemite 10.10.5 here, too.

I have the Install apps from anywhere setting. It used to work without problem until this latest build. I found several posts about this error and all of them suggest allowing installation of apps from anywhere. I have that. This SO answer points to something else.

[giumas]

@ajelenak-thg : I apologize in advance for the silly question, but did you delete first any previous installation? If you freeze the app on your machine, does it work?

[tedhabermann]

I just downloaded the link That John used on Yosemite 10.10.5 and it appears to be working as expected.

Are there good geospatial datasets to try the mapping additions?

Ted

[giumas]

I usually use this small size file for testing: https://bitbucket.org/ccomjhc/hyo_bag/downloads/W00275_MB_50cm_MLLW_1of1.bag, but any of the BAG files present on the NOAA Bathymetric Data Viewer should work fine: http://maps.ngdc.noaa.gov/viewers/bathymetry/

[ghost]

@giumas

I apologize in advance for the silly question

There are no silly questions at this point... :-)

but did you delete first any previous installation?

Yes, tried several times, too.

If you freeze the app on your machine, does it work?

Never done that before so don't have the required toolset.

Don't want to make big deal over this. It's okay if I am the only Mac user in the world who cannot run HDF Compass. :-)

[giumas]

@ajelenak-thg : here you can find the step-by-step instructions: https://github.com/HDFGroup/hdf-compass/wiki/Notes-on-freezing-with-Mac-OS-X

[jreadey]

I'm not sure @ajelenak-thg following the step by step instructions would be very informative. If there is some missing dependency with the install image, I'd rather resolve that so everyone can run the install image.

@ajelenak-thg - Are you able to run the dmg installer? If so, can you runn HDFCompass from the command line? (In shell, cd to: /Applications/HDFCompass.app/Contents/MacOS, and run ./HDFCompass_0.6.0b6)

[ghost]

@ajelenak-thg - Are you able to run the dmg installer?

Yes. The verification when mounting the dmg passes.

If so, can you runn HDFCompass from the command line? (In shell, cd to: /Applications/HDFCompass.app/Contents/MacOS, and run ./HDFCompass_0.6.0b6)

Yes, I can. This is the startup trace displayed in the terminal window:

PyInstaller Bootloader 3.x
LOADER: executable is /Applications/HDFCompass.app/Contents/MacOS/HDFCompass_0.6.0b6
LOADER: homepath is /Applications/HDFCompass.app/Contents/MacOS
LOADER: _MEIPASS2 is NULL
LOADER: archivename is /Applications/HDFCompass.app/Contents/MacOS/HDFCompass_0.6.0b6
LOADER: Extracting binaries
LOADER: Executing self as child
LOADER: set _MEIPASS2 to /Applications/HDFCompass.app/Contents/MacOS
PyInstaller Bootloader 3.x
LOADER: executable is /Applications/HDFCompass.app/Contents/MacOS/HDFCompass_0.6.0b6
LOADER: homepath is /Applications/HDFCompass.app/Contents/MacOS
LOADER: _MEIPASS2 is /Applications/HDFCompass.app/Contents/MacOS
LOADER: archivename is /Applications/HDFCompass.app/Contents/MacOS/HDFCompass_0.6.0b6
LOADER: Already in the child - running user's code.
LOADER: Python library: /Applications/HDFCompass.app/Contents/MacOS/Python
LOADER: Loaded functions from Python library.
LOADER: Manipulating environment (sys.path, sys.prefix)
LOADER: sys.prefix is /Applications/HDFCompass.app/Contents/MacOS
LOADER: Setting runtime options
LOADER: Initializing python
LOADER: Overriding Python's sys.path
LOADER: Post-init sys.path is /Applications/HDFCompass.app/Contents/MacOS
LOADER: Setting sys.argv
LOADER: setting sys._MEIPASS
LOADER: importing modules from CArchive
LOADER: extracted struct
LOADER: callfunction returned...
LOADER: extracted pyimod01_os_path
LOADER: callfunction returned...
LOADER: extracted pyimod02_archive
LOADER: callfunction returned...
LOADER: extracted pyimod03_importers
LOADER: callfunction returned...
LOADER: Installing PYZ archive with Python modules.
LOADER: PYZ archive: out00-PYZ.pyz
LOADER: Running pyiboot01_bootstrap.py
LOADER: Running pyi_rth_pkgres.py
LOADER: Running pyi_rth__tkinter.py
LOADER: Running pyi_rth_mplconfig.py
LOADER: Running pyi_rth_mpldata.py
LOADER: Running HDFCompass.py
/Applications/HDFCompass.app/Contents/MacOS/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
Trying `CDLL(/Applications/HDFCompass.app/Contents/MacOS/HDFCompass_0.6.0b6/libgeos_c.dylib)`
DEBUG   shapely.geos.load_dll:46 > Trying `CDLL(/Applications/HDFCompass.app/Contents/MacOS/HDFCompass_0.6.0b6/libgeos_c.dylib)`
Failed `CDLL(/Applications/HDFCompass.app/Contents/MacOS/HDFCompass_0.6.0b6/libgeos_c.dylib)`
WARNING shapely.geos.load_dll:50 > Failed `CDLL(/Applications/HDFCompass.app/Contents/MacOS/HDFCompass_0.6.0b6/libgeos_c.dylib)`
Trying `CDLL(/Applications/HDFCompass.app/Contents/MacOS/libgeos_c.1.dylib)`
DEBUG   shapely.geos.load_dll:46 > Trying `CDLL(/Applications/HDFCompass.app/Contents/MacOS/libgeos_c.1.dylib)`
Library path: '/Applications/HDFCompass.app/Contents/MacOS/libgeos_c.1.dylib'
DEBUG   shapely.geos.load_dll:54 > Library path: '/Applications/HDFCompass.app/Contents/MacOS/libgeos_c.1.dylib'
DLL: <PyInstallerCDLL '/Applications/HDFCompass.app/Contents/MacOS/libgeos_c.1.dylib', handle 7feff3b398f0 at 10b7b1650>
DEBUG   shapely.geos.load_dll:55 > DLL: <PyInstallerCDLL '/Applications/HDFCompass.app/Contents/MacOS/libgeos_c.1.dylib', handle 7feff3b398f0 at 10b7b1650>
Trying `CDLL(/usr/lib/libc.dylib)`
DEBUG   shapely.geos.load_dll:37 > Trying `CDLL(/usr/lib/libc.dylib)`
Library path: '/usr/lib/libc.dylib'
DEBUG   shapely.geos.load_dll:54 > Library path: '/usr/lib/libc.dylib'
DLL: <PyInstallerCDLL '/usr/lib/libc.dylib', handle 7fff64311830 at 10b7b1610>
DEBUG   shapely.geos.load_dll:55 > DLL: <PyInstallerCDLL '/usr/lib/libc.dylib', handle 7fff64311830 at 10b7b1610>
DEBUG   hdf_compass.compass_viewer.viewer.load_plugins:190 > Python 2.7.11 64bit on Darwin 14.5.0 (x86_64)
DEBUG   hdf_compass.compass_viewer.viewer.load_plugins:192 > numpy 1.11.0
DEBUG   hdf_compass.compass_viewer.viewer.load_plugins:193 > matplotlib 1.5.1
DEBUG   hdf_compass.compass_viewer.viewer.load_plugins:194 > wxPython 3.0.2.0
DEBUG   hdf_compass.compass_viewer.viewer.load_plugins:211 > h5py 2.6.0
DEBUG   hdf_compass.compass_viewer.viewer.load_plugins:219 > hydroffice.bag 0.2.12
DEBUG   hdf_compass.compass_viewer.viewer.load_plugins:222 > lxml 3.6.0 (libxml 2.9.0, libxslt 1.1.28)
DEBUG   hdf_compass.compass_viewer.viewer.load_plugins:235 > pydap 3.1.1 (protocol 2.15)
DEBUG   hdf_compass.compass_viewer.frame.__init__:71 > new frame -> open frames: 1

[tedhabermann]

All,

Just used compass on the mac to look at a BAG... It was very pleasant. Will demo to IHO this week...

The BAG viewer has an XML view for the ISO metadata... SMAP also has a big blob of ISO - would it be hard to make that viewer work for the SMAP metadata?

Ted

==== Ted Habermann ========================
Director of Earth Science, The HDF Group
Voice: (217) 531-4202
Email: [email protected]
==== HDF: Software That Powers Science ====

[giumas]

Here we go: https://pypi.python.org/pypi/hdf_compass/0.6.0
I would propose to open a new ticket with the milestone for 0.7.0

[jreadey]

"Official" releases for Windows and Mac are on s3 now:

OS X: http://hdfgroup.download.s3.amazonaws.com/HDFCompass/0.6.0/HDFCompass.dmg
Windows: http://hdfgroup.download.s3.amazonaws.com/HDFCompass/0.6.0/HDFCompass.exe

I verified that plotting and the BAG plugin works correctly and that the version is correct. If anyone spots other issues, please let me know.

[giumas]

@jreadey given the ticket for 0.7.0, can we close this one?

[jreadey]

Yes - closing.