test_urlopen fails recently

[olebole]

Hi,

since some time, the test_urlopen test of this package fail during the build for Debian. When I built the latest version of asdf for Debian in November 2017, all tests went fine, however. So, this is probably connected to some changes in our (build) environment; but I have no idea which this could be.

The output is

_____________________________ test_urlopen[tree0] ______________________________

tree = {'not_shared': array([10,  9,  8,  7,  6,  5,  4,  3,  2,  1], dtype=uint8), 'science_data': array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.]), 'skipping': array([ 0.,  2.,  4.,  6.,  8.]), 'subset': array([ 3.,  4.,  5.,  6.])}
httpserver = <asdf.conftest.HTTPServer object at 0x7f353b5fdb50>

    @pytest.mark.skipif(INTERNET_OFF, reason="Astropy has disabled internet access")
    @pytest.mark.skipif(sys.platform.startswith('win'),
                        reason="Windows firewall prevents test")
    def test_urlopen(tree, httpserver):
        path = os.path.join(httpserver.tmpdir, 'test.asdf')
    
        def get_write_fd():
            return generic_io.get_file(open(path, 'wb'), mode='w')
    
        def get_read_fd():
            return generic_io.get_file(
                urllib_request.urlopen(
                    httpserver.url + "test.asdf"))
    
>       with _roundtrip(tree, get_write_fd, get_read_fd) as ff:

asdf/tests/test_generic_io.py:270: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
asdf/tests/test_generic_io.py:72: in _roundtrip
    with get_read_fd() as fd:
asdf/tests/test_generic_io.py:268: in get_read_fd
    httpserver.url + "test.asdf"))
/usr/lib/python2.7/urllib2.py:154: in urlopen
    return opener.open(url, data, timeout)
/usr/lib/python2.7/urllib2.py:429: in open
    response = self._open(req, data)
/usr/lib/python2.7/urllib2.py:447: in _open
    '_open', req)
/usr/lib/python2.7/urllib2.py:407: in _call_chain
    result = func(*args)
/usr/lib/python2.7/urllib2.py:1228: in http_open
    return self.do_open(httplib.HTTPConnection, req)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib2.HTTPHandler instance at 0x7f353b4701b8>
http_class = <class httplib.HTTPConnection at 0x7f3546f6f3f8>
req = <urllib2.Request instance at 0x7f353b4c8e18>, http_conn_args = {}
host = '127.0.0.1:9', h = <httplib.HTTPConnection instance at 0x7f353b46fb00>
err = error(111, 'Connection refused')

    def do_open(self, http_class, req, **http_conn_args):
        """[...]"""
        host = req.get_host()
        if not host:
            raise URLError('no host given')
    
        # will parse host:port
        h = http_class(host, timeout=req.timeout, **http_conn_args)
        h.set_debuglevel(self._debuglevel)
    
        headers = dict(req.unredirected_hdrs)
        headers.update(dict((k, v) for k, v in req.headers.items()
                            if k not in headers))
    
        # We want to make an HTTP/1.1 request, [...]
        headers["Connection"] = "close"
        headers = dict(
            (name.title(), val) for name, val in headers.items())
    
        if req._tunnel_host:
            tunnel_headers = {}
            proxy_auth_hdr = "Proxy-Authorization"
            if proxy_auth_hdr in headers:
                tunnel_headers[proxy_auth_hdr] = headers[proxy_auth_hdr]
                # Proxy-Authorization should not be sent to origin
                # server.
                del headers[proxy_auth_hdr]
            h.set_tunnel(req._tunnel_host, headers=tunnel_headers)
    
        try:
            h.request(req.get_method(), req.get_selector(), req.data, headers)
        except socket.error, err: # XXX what error?
            h.close()
>           raise URLError(err)
E           URLError: <urlopen error [Errno 111] Connection refused>

/usr/lib/python2.7/urllib2.py:1198: URLError

Same is for test_urlopen[tree1]. Also, there is failure of test_internet_off:

______________________________ test_internet_off _______________________________

    def test_internet_off():
        if not _REMOTE_DATA:
>           assert INTERNET_OFF == True
E           assert False == True

asdf/tests/test_remote_data.py:22: AssertionError

This is also reported as Debian bug #887531.

[drdavella]

Hi @olebole, thanks for the bug report. I'll try to spin up a debian VM in the near future to see if I can reproduce these issues.

[olebole]

Thanks. Please keep in mind that during the build, remote data access is disabled.

[drdavella]

@olebole I just released v1.3.2, which should resolve this issue. Please reopen if you encounter it again.

[olebole]

This is still not fixed, sorry. The second error (AssertionError in test_internet_off) is gone, but test_urlopen() still fails. I can't however reopen the issue, pls do it yourself.

[drdavella]

Can you please post the entire test output log here, along with steps to reproduce? I have tested this on a Debian system but obviously there is something different in our environments.

[olebole]

Sure...
python-asdf_1.3.2-1_amd64.txt

[drdavella]

Okay, I've got a hunch about this one now, but I'm still not able to reproduce #447.

[drdavella]

@olebole this should now be fixed on stable. I'm about to issue a new release.