Clip exponential in ffunc to avoid overflow

[zuphilip]

This should avoid (hopefully) some possible FloatingPointError overflow errors.

The sigmoid function ffunc is for any x<-20 and x>20 already 0 resp. 1 up to 10^-9
and cutting will therefore not change the function substantially.

This idea is from @tmbdev in #5 (comment)
Implemented first in #49 (comment)
Additional infos from #79 (comment)

[zuphilip]

Note that some lines above we have:

def sigmoid(x):
    """Compute the sigmoid function.
    We don't bother with clipping the input value because IEEE floating
    point behaves reasonably with this function even for infinities."""
return 1.0/(1.0+exp(-x))

• Do we need both identical functions ffunc and sigmoid?
• What are "IEEE floating points"?
• Should we reconsider the comment here?

[mittagessen]

-20/20 seems to be overly conservative but should work. Ultimately the gradients are important and they are negligible even at -10/10. On another note scipy.special.expit is significantly faster than .0/(1.0+exp(-x)), so you might consider using that.

You won't need both ffunc and sigmoid as pickling only retains references to classes and function. Adjusting the forward/backward methods of the appropriate classes should be enough.

IEEE floating points are standard IEEE 754 floating point numbers. The comment suggests they are stable even for extreme values. The linked bug reports seem to contradict that notion.

[kba]

I'd like to merge it, and/or try @mittagessen's proposed scipy fix. @zuphilip do you have data to test this?

[zuphilip]

@kba I think it is good to merge this. Maybe reformulate the contradicting comment before... I don't have data for reproduce the error, but there is at least one confirmation #246 (comment) that after this fix it worked fine.

[kba]

Yes, this will fix a breaking bug, so let's merge it. Just would be interesting to test scipy.special.expit. But working is better than fast.

[zuphilip]

@mittagessen and @kba Thank you for helping here!

[amitdo]

https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.expit.html#scipy.special.expit

@mittagessen mentioned this function as a replacement to sigmoid/ffunc.