Functions that run in Julia can raise MethodError in Python

[MatthewZalesak]

The simplest example I could find of this occurs while running the Julia package "LowRankModels" in Python (not in Julia). I suspect this is a problem with "pyjulia" rather than "LowRankModels" since the error only occurs while running a test file in Python rather when running the same test in Julia.

In short, there is a function fit!(glrm) that runs in Julia but the Python call j.fit_b(glrm) raises a MethodError on what should be the same computation.

Setup

• Ubuntu Docker container (the error also occurs in Fedora)
• Most recent version of Julia (0.6) installed from the Julia website
• Python 2.7
• Julia packages PyCall, LowRankModels, FactCheck, NullableArrays
• Pip installed numpy, scipy, pyjulia
• Pull latest git commit on PyCall and LowRankModels, then rebuilt

Code Leading to Error

The Python test file is at hello_world.py. The relevant snippet is

from numpy.random import rand, randn, choice
from numpy import hstack

import julia
j = julia.Julia()
j.using("LowRankModels")

real_loss_types = [j.QuadLoss, j.HuberLoss]
bool_loss_types = [j.HingeLoss]
ordinal_loss_types = [j.OrdinalHingeLoss, j.BvSLoss]
categorical_loss_types = [j.MultinomialLoss, j.OvALoss]

#instantiate losses
ncat = 4
nord = 5
real_losses = [l() for l in real_loss_types]
bool_losses = [l() for l in bool_loss_types]
ordinal_losses = [l(choice(range(3,nord))) for l in ordinal_loss_types]
categorical_losses = [l(choice(range(3,ncat))) for l in categorical_loss_types]
losses = real_losses + bool_losses + ordinal_losses + categorical_losses
for loss in losses:
    j.scale_b(loss, rand()) # scale_b is the julia function scale!
regularizers = [j.QuadReg(), j.OneReg(5), j.NonNegConstraint(), j.KSparseConstraint(2)]

m = len(regularizers)
n = len(losses)

A_real = rand(m, len(real_losses))
A_bool = choice([True, False], size = (m, len(bool_losses)))
A_ord = choice(range(1,6), size = (m, len(ordinal_losses))) # use 1 indexing
A_cat = choice(range(1,4), size = (m, len(categorical_losses))) # use 1 indexing
A = hstack([A_real, A_bool, A_ord, A_cat])

glrm = j.GLRM(A, losses, regularizers, j.QuadReg(), 2) 
j.fit_b(glrm) # fit_b is the julia function fit!

Running in Python, an error occurs while handling the function j.fit_b(glrm) on line 53, the last line. The error states

RuntimeError: Julia exception: MethodError(LowRankModels.evaluate, (LowRankModels.WeightedHingeLoss(0.7345470143907986, LowRankModels.BoolDomain(), 1.0), 0.939190790821008, 1.0), 0x000000000000554e)

The function being called in Julia is fit!(glrm), or j.fit_b(glrm) in the Python script, so the error which is identified as LowRankModels.evaluate seems to be coming from a subcall within Julia.

The equivalent test file in Julia is at hello_world.jl. The snippet is

using LowRankModels

real_loss_types = [QuadLoss, HuberLoss]
bool_loss_types = [HingeLoss]
ordinal_loss_types = [OrdinalHingeLoss, BvSLoss]
categorical_loss_types = [MultinomialLoss, OvALoss]

# instantiate losses
ncat = 4
nord = 5
real_losses = [l() for l in real_loss_types]
bool_losses = [l() for l in bool_loss_types]
ordinal_losses = [l(rand(3:nord)) for l in ordinal_loss_types]
categorical_losses = [l(rand(3:ncat)) for l in categorical_loss_types]
losses = [real_losses..., bool_losses..., ordinal_losses..., categorical_losses...]
for loss in losses
    scale!(loss, rand())
end
regularizers = [QuadReg(), OneReg(5), NonNegConstraint(), KSparseConstraint(2)]
m,n = length(regularizers), length(losses)

A_real = rand(m, length(real_losses))
A_bool = rand(Bool, m, length(bool_losses))
A_ord = rand(1:5, m, length(ordinal_losses))
A_cat = rand(1:3, m, length(categorical_losses))
A = Any[A_real A_bool A_ord A_cat]

glrm = GLRM(A, losses, regularizers, QuadReg(), 2)
fit!(glrm)

When it runs the same function call, fit!(glrm), no error is produced.

Any ideas about what could be causing this error would be appreciated.

[madeleineudell]

I know the origin of this problem. The last argument of evaluate(WeightedHingeLoss(), .95, 1) needs to be an integer; instead, we are accidentally passing a floating point number (1.), resulting in a MethodError. This happens because the array A ends up being a floating point array, rather than having different columns have different types (the numbers get up-converted).

I can fix this on the Julia side in LowRankModels by making the types looser. But here are two possible lessons for pyjulia:

1. the error message is a bit short; perhaps it would be better to see the full stack trace?
2. where python and julia differ in procedures for conversions between types, we're bound to see more errors like this. is there any way to coerce the resulting calls to use the number type for which the corresponding method exists?

[tkf]

It seems @madeleineudell provided an answer already? Please reopen if you still have the issue.

the error message is a bit short; perhaps it would be better to see the full stack trace?

JuliaPy/PyCall.jl#608 solved this.