python3 porting?

[filefolder]

Hi, our webserver (and seiscomp instance) is now exclusively python3 which breaks webdc3. Anyone managed to successfully port over? For the most part something like 2to3 catches most of the changes but there are still a lot of encoding gremlins left over.

I'm particularly having trouble with the end of webinterface.py

    if isinstance(res_string, basestring):
        status = '200 OK'
        body = res_string
        return send_plain_response(status, body, start_response)

    elif hasattr(res_string, 'filename'):
        status = '200 OK'
        body = res_string
        return send_file_response(status, body, start_response)

    status = '200 OK'
    body = "\n".join(res_string)
    return send_plain_response(status, body, start_response)

I understand that 'basestring' should just be changed to 'str' in py3, but at that point the code won't run unless I encode the body variable in send_plain_response in wsgicomm.py (e.g. body = body.encode('utf-8')), but this then ends up breaking other stuff so I am thinking there is something else here.

Anyway, curious if someone has already done this and know a fix, and if not, fwiw, the demand is there. Thanks!

[Fran89]

Hello there,

You aren't alone in this. Same problem over here, managed to fix ones for update-metadata and still find encoding gremlins all over, specially with urllib. I'm currently cleaning up the inventory loader. This will take a bit of time and patience. Any help from the devs would be most helpful.

[filefolder]

Hi, great to hear others are still interested.

FWIW, I've managed to hack together a (mostly) working version (wdc3fixes.zip --forgive me not branching them properly) with most of my edits flagged with "RCP", I hope it helps anyone else stuck somewhere. However I still have issues when downloading anything, with the console just listing the metadata and the word ERROR next to each line in red but I've had to quit working on it for the moment.

[Fran89]

I've been thinking of using the https://github.com/EIDA/eida-portal which is a FDSNWS front-end that's a bit friendlier to use. What do you think?

[filefolder]

Ah I was unaware of this, although it seems functionally equivalent to webdc3 (?) I wonder if it would be easier to get working. At the moment it is probably more economical for me to fix the last few errors here rather than set up a whole new program.

There is also https://github.com/iris-edu/pyweed, but it is a stand alone code and it does not allow one to customise the FDSN server list unfortunately which is a non-starter for us.

[Fran89]

It seems to be functionally equivalent. It doesn't seem to hard to edit, but it requires a bit of reverse engineering but I managed to do it, and after it's all said and done I get a docker image I can deploy.

[javiquinte]

Dear @Fran89, dear @filefolder ,

First of all, thanks for using webdc3 and the interest in its update! I'm writing to you both as part of the development team to clarify a couple of points. You know that this software has been developed some years ago and we have updated it regularly from the point of view of the functionality. Originally worked with Arclink and later with the standard fdsn dataselect and stations web services. It can also run as stand alone package for a data centre or behind a federation like EIDA.
However, we postponed the migration to python3 because some colleagues from us at KNMI/ODC were working on a new development which could supersede webdc3.
We are currently discussing if the migrating effort makes sense or not, because if the new development will be released soon and it is as flexible as webdc3, we would prefer to wait for that.
Otherwise, we will evaluate how and when to upgrade webdc3.

We will get back to you soon when we have a clear roadmap for this.

Thank again for the ideas and the efforts already posted regarding the migration of this software!

Javier (on behalf of the GEOFON team)

[Fran89]

Dear @javiquinte, @filefolder,

That's great I would love to see this project updated and ready to go because it was so relatively easy to setup. If the superceded version is similar to EIDA-Portal that would be awesome too, it's a pretty mature project that just requires a bit more documentation and an easier way to setup. Making some changes I did manage to change my WebDC3 to EIDA Portal.

https://sc.smp.uprm.edu/smp/

I just wish I could easily customize it further like I was able to with WebDC3. (Change some default settings and parameters, make debugging made simpler). I ran into a few things that were weird and will make an issue ticket over there.

Got off track, the idea is that if you would update it I'd like to switch back to WebDC3, but at this point I would just like to have a good customizable FDSNWS web interface.

Thank you for maintaining software like this!

Fran89 (PRSMP)

[javiquinte]

Dear @filefolder , dear @Fran89 ,
we finally have discussed internally and decided that we will update Webdc3 to the latest version of everything. This is definitely not our priority, so it could take from a couple of weeks to a couple of months (hopefully weeks).

Thanks again to you for the patience and the support. We'll announce the advances and probably contact you in case of early releases.

[wwxuww]

Dear @javiquinte @Fran89 @filefolder ,
Glad to find your posting here, since I am running into the very similar situation. After deploying the Seiscomp4.8.4 successfully I am doing WebDC3 deployment and also similarly with urllib-like problem and stuck here.
May I ask if you had finished the webdc3 update program or do you have some other better suggestion or advice? Thanks a lot!
Wei

[filefolder]

I was just thinking about how this was going as I found myself having to hack another fix to keep this running last week. Off the top of my head there are a few important edits going from sc3 to sc4

1. many/all of the seiscomp imports have to be changed to "seiscomp3" e.g. from seiscomp3.xmlparser import DateTimeAttr

2. replace all cgi module calls with html

3. replace all the old urllib2 calls with these: import urllib.request, urllib.error, urllib.parse

4. there are many instances where temp files need to be read/written with 'wt' instead of 'wb'

There are still many binary encoding issues I am forgetting (and likely have yet to find) and I can't actually get FDSNWS event servers to work properly (datetimes are binary encoded and I can't figure out where to decode them), only the geofon server works.

As an alternative, the non-web based "pyweed" from IRIS is nice and I would love to see webdc3 offer waveform visualisation in a similar way, but it still has a lot of serious issues and unfortunately appears to be abandoned.

[pevans-gfz]

Thanks for your suggestions. I got halfway through this a few weeks ago now, but not quite to the end. I hope to push something soon.

[filefolder]

that's great to hear. if you are taking suggestions then some form of queryauth capability to access restricted data would be much appreciated.