Cython + 'unexpected keyword argument' hack

[nilgoyette]

I'm not sure if Luigi or Cython should be blamed, but here's the problem I found. As you know, we can find this snippet in worker.py in _run_get_new_deps:

try:
    task_gen = self.task.run(tracking_url_callback=self.tracking_url_callback)
except TypeError as ex:
    if 'unexpected keyword argument' not in str(ex):
        raise

Although it's clearly a hack, it does work most of the time. The exception message in indeed: got an unexpected keyword argument 'tracking_url_callback"

When we cythonize our program though, the exception message becomes: run() takes no keyword arguments, which makes Luigi unable to do anything.

I could push a quick&dirty patch, but I prefer asking before because you might prefer to get rid of this hack instead.

[Tarrasch]

In recent versions of luigi we removed this hack, but let it still be there for keeping backward comptibility for this short-lived hack. See https://github.com/spotify/luigi/pull/1625/files#diff-b438c8dc80f50172445c72006dc7c8f7R121

Hmm. It's been two deprecated for two months now. Usually that's not enough, but as this such a terrible hack and it causes Cython to not work. I'll be happy to accept a PR removing this hack.

In addition, you should perhaps add so that travis also runs the Cython tests, if you don't want it to break unexpectedly again :)

[nilgoyette]

I started reading about travis/ox and I think I can do what you ask, but I have a problem with the cython part. I thought that cythonizing a project was as simple as calling python setup.py build_ext but it does nothing. Is there currently a way to cythonize luigi?

[Tarrasch]

The only thing I know about cython is that it's something like C + Python. I've never used it. Sorry :)

[nilgoyette]

I cythonized most of the project (some files, like init.py, main.py, six.py, etc., are best ignored). It's not terribly complicated but it took hours to do because I never did it before. The problem is that I now have all kinds of weird errors like:
PicklingError: Can't pickle <class 'task.Task'>: it's not the same object as task.Task
Task.bulk_complete at 0x7f99b28f2590> is not a Python function
and a some totally unrelated tests fail.

Frankly, I don't have the time (and the motivation) to continue. And the company I work for only care about the hack above, not the cython part. So, my question is, will you accept the pull request if it contains only the removal of the hack?

(I could add a test that only cythonize a single file (worker.py) and test if everything is ok. I just doubt that this test will be useful to anyone)

[Tarrasch]

Thanks for your efforts. Please at least push your work-in-progress branch and link to it from this issue. In case it will be useful for anybody.

So, my question is, will you accept the pull request if it contains only the removal of the hack?

Yes. Absolutely.

[nilgoyette]

As requested, I commited my cython tests on a branch on my fork.
https://github.com/nilgoyyou/luigi/tree/cython-tests