segfault with large data #71

messense · 2017-07-31T06:43:10Z

I have a function with signature:

#[pyfn(m, "extract_entity")]
fn extract_entity_py(py: Python, content: Vec<Vec<String>>) -> PyResult<Vec<Vec<Vec<Entity>>>> {}

with large data it segfaults (small data is fine), it happens on Linux, on macOS it's mostly fine.

dmesg log:

[262892.915631] pytest[17095]: segfault at 8b1 ip 0000000000454295 sp 00007ffc02678980 error 4 in python3.6[400000+26e000]
[262928.148360] pytest[17275]: segfault at 7f20212fbd58 ip 00007f20212fbd58 sp 00007ffc94a31de8 error 15 in libc-2.17.so[7f20212fb000+2000]
[262939.497128] pytest[17385]: segfault at 7ff0a03e9d58 ip 00007ff0a03e9d58 sp 00007ffd8be92d78 error 15 in libc-2.17.so[7ff0a03e9000+2000]

The text was updated successfully, but these errors were encountered:

messense · 2017-07-31T06:45:50Z

It happens on both Python 2.7 and Python 3.6

fafhrd91 · 2017-07-31T07:00:26Z

Could you post backtrace

messense · 2017-07-31T07:01:03Z

Couldn't get any backtrace. RUST_BACKTRACE=1 gives nothing.

messense · 2017-07-31T07:02:11Z

#[pyfn(m, "extract_entity")]
fn extract_entity_py(py: Python, content: Vec<Vec<String>>) -> PyResult<Vec<Vec<Vec<Entity>>>> {
    println!("{:?}", content);
}

the println!("{:?}", content); doesn't even get executed.

fafhrd91 · 2017-07-31T07:08:37Z

Try to run it with lldb

messense · 2017-07-31T07:18:09Z

During startup program terminated with signal SIGSEGV, Segmentation fault.

(gdb) bt
No stack.

fafhrd91 · 2017-07-31T07:32:39Z

Seems like something wrong with executable. Are you sure you build it with right arch?

…

Sent from my iPhone

On Jul 31, 2017, at 12:18 AM, messense ***@***.***> wrote: During startup program terminated with signal SIGSEGV, Segmentation fault. (gdb) bt No stack. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

messense · 2017-07-31T07:33:30Z

I am sure. It works fine with small amount of data.

messense · 2017-07-31T07:42:56Z

Reading symbols from python...Reading symbols from /usr/lib/debug/.build-id/33/8517c6b7aeca89dd84881d97731824404a3223.debug...done.
done.

warning: core file may not match specified executable file.
[New LWP 434]
[New LWP 439]
[New LWP 441]
[New LWP 440]
[New LWP 442]
[New LWP 438]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `python article_segment.py'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000459755 in PyIter_Next (iter=0x21f56e0) at ../Objects/abstract.c:3147
3147	../Objects/abstract.c: No such file or directory.
[Current thread is 1 (Thread 0x7f7d23a8e700 (LWP 434))]

Got some backtrace with core dump.

fafhrd91 · 2017-07-31T07:43:00Z

Could you post reproducible code

messense · 2017-07-31T07:46:19Z

It'll take some time to extract reproducible code. More bt:

#0  0x0000000000459755 in PyIter_Next (iter=<unknown at remote 0x21f56e0>) at ../Objects/abstract.c:3147
#1  0x00007f7d1e4b9e3e in pyo3::objects::iterator::{{impl}}::next (self=0x7ffe2f5b19b8) at /root/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/pyo3-0.1.0/src/objects/iterator.rs:48
#2  0x00007f7d1e289a63 in pyo3::objects::sequence::extract_sequence<alloc::vec::Vec<alloc::string::String>> (obj=0x202ce18) at /root/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/pyo3-0.1.0/src/objects/sequence.rs:237
#3  0x00007f7d1e28a934 in pyo3::objects::sequence::{{impl}}::extract<alloc::vec::Vec<alloc::string::String>> (obj=0x202ce18) at /root/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/pyo3-0.1.0/src/objects/sequence.rs:210
#4  0x00007f7d1e29028c in pyo3::objectprotocol::{{impl}}::extract<pyo3::objects::PyObjectRef,alloc::vec::Vec<alloc::vec::Vec<alloc::string::String>>> (self=0x202ce18) at /root/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/pyo3-0.1.0/src/objectprotocol.rs:367
#5  0x00007f7d1e2bc35b in litigant::init_module::__wrap (_slf=0x0, _args=0x7f7d22644438, _kwargs=0x0) at src/lib.rs:242
#6  0x00000000004b295c in _PyCFunction_FastCallDict (kwargs=0x0, nargs=<optimized out>, args=0x7f7d10002168, func_obj=<built-in function extract_entity>) at ../Objects/methodobject.c:231
#7  _PyCFunction_FastCallKeywords (func=func@entry=<built-in function extract_entity>, stack=stack@entry=0x7f7d10002168, nargs=<optimized out>, kwnames=kwnames@entry=0x0) at ../Objects/methodobject.c:294
#8  0x00000000005434ae in call_function (pp_stack=pp_stack@entry=0x7ffe2f5b21a0, oparg=oparg@entry=1, kwnames=kwnames@entry=0x0) at ../Python/ceval.c:4809
#9  0x00000000005454bd in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3295
Python Exception <class 'RuntimeError'> Type does not have a target.:
#10 0x0000000000542771 in PyEval_EvalFrameEx (throwflag=0, f=) at ../Python/ceval.c:718
Python Exception <class 'RuntimeError'> Type does not have a target.:
#11 _PyFunction_FastCall (co=<optimized out>, args=<optimized out>, nargs=1, globals=globals@entry=) at ../Python/ceval.c:4891
#12 0x0000000000543621 in fast_function (kwnames=0x0, nargs=<optimized out>, stack=<optimized out>, func=<function at remote 0x7f7d1e84abf8>) at ../Python/ceval.c:4926
#13 call_function (pp_stack=pp_stack@entry=0x7ffe2f5b2350, oparg=oparg@entry=1, kwnames=kwnames@entry=0x0) at ../Python/ceval.c:4830
#14 0x00000000005454bd in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3295
Python Exception <class 'RuntimeError'> Type does not have a target.:
#15 0x0000000000542771 in PyEval_EvalFrameEx (throwflag=0, f=) at ../Python/ceval.c:718
Python Exception <class 'RuntimeError'> Type does not have a target.:
#16 _PyFunction_FastCall (co=<optimized out>, args=<optimized out>, nargs=1, globals=globals@entry=) at ../Python/ceval.c:4891
#17 0x0000000000543621 in fast_function (kwnames=0x0, nargs=<optimized out>, stack=<optimized out>, func=<function at remote 0x7f7d1e8590d0>) at ../Python/ceval.c:4926
#18 call_function (pp_stack=pp_stack@entry=0x7ffe2f5b2500, oparg=oparg@entry=1, kwnames=kwnames@entry=0x0) at ../Python/ceval.c:4830
#19 0x00000000005454bd in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3295
Python Exception <class 'RuntimeError'> Type does not have a target.:
#20 0x0000000000543115 in PyEval_EvalFrameEx (throwflag=0, f=) at ../Python/ceval.c:718
#21 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7f7d226ad8a0>, globals=globals@entry=< at remote 0x7f7d2270d270>, locals=locals@entry=<code at remote 0x7f7d226ad8a0>, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0, kwargs=0x8,
    kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at ../Python/ceval.c:4139
#22 0x0000000000543f43 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, locals=locals@entry=<code at remote 0x7f7d226ad8a0>, globals=globals@entry=< at remote 0x7f7d2270d270>,
    _co=_co@entry=<code at remote 0x7f7d226ad8a0>) at ../Python/ceval.c:4160
Python Exception <class 'RuntimeError'> Type does not have a target.:
Python Exception <class 'RuntimeError'> Type does not have a target.:
#23 PyEval_EvalCode (co=co@entry=<code at remote 0x7f7d226ad8a0>, globals=globals@entry=, locals=locals@entry=) at ../Python/ceval.c:695
Python Exception <class 'RuntimeError'> Type does not have a target.:
Python Exception <class 'RuntimeError'> Type does not have a target.:
Python Exception <class 'RuntimeError'> Type does not have a target.:
#24 0x000000000042ac8f in run_mod (arena=0x7f7d2270d270, flags=0x7ffe2f5b27e0, locals=, globals=, filename=, mod=0x1c21208) at ../Python/pythonrun.c:980
Python Exception <class 'RuntimeError'> Type does not have a target.:
Python Exception <class 'RuntimeError'> Type does not have a target.:
#25 PyRun_FileExFlags (fp=0x1ba9d70, filename_str=<optimized out>, start=<optimized out>, globals=, locals=, closeit=1, flags=0x7ffe2f5b27e0) at ../Python/pythonrun.c:933
#26 0x000000000042aebc in PyRun_SimpleFileExFlags (fp=0x1ba9d70, filename=<optimized out>, closeit=1, flags=0x7ffe2f5b27e0) at ../Python/pythonrun.c:396
#27 0x000000000043f1f5 in run_file (p_cf=0x7ffe2f5b27e0, filename=0x1b67260 L"article_segment.py", fp=0x1ba9d70) at ../Modules/main.c:338
#28 Py_Main (argc=argc@entry=2, argv=argv@entry=0x1b66010) at ../Modules/main.c:809
#29 0x00000000004213d2 in main (argc=2, argv=<optimized out>) at ../Programs/python.c:69

fafhrd91 · 2017-07-31T07:47:39Z

Do you use master or 0.1 release?

messense · 2017-07-31T07:48:17Z

Tried both, they all have the same issue, the backtrace is generated from 0.1 release

fafhrd91 · 2017-07-31T07:52:27Z

Ok. Try to extract reproducible code

messense · 2017-07-31T08:24:21Z

@fafhrd91 I have sent you the code through email.

messense · 2017-07-31T08:46:23Z

https://github.com/PyO3/pyo3/blob/master/src/objects/iterator.rs#L47

Change:

py.cast_from_ptr_or_opt(ffi::PyIter_Next(self.0.as_ptr())) }

to

py.cast_from_borrowed_ptr_or_opt(ffi::PyIter_Next(self.0.as_ptr())) }

fixed the segfault on linux(and still works on macOS)

fafhrd91 · 2017-08-01T21:10:10Z

I added temp fix. could you try to run your code with English text?

messense · 2017-08-02T01:40:30Z

Confirmed the fix with both Chinese text and English text.

fafhrd91 · 2017-08-02T01:44:09Z

But does it segfault with English without latest fix?

messense · 2017-08-02T01:44:55Z

It does, but it needs more data to trigger.

messense · 2017-08-02T01:47:13Z

I used the Zen of Python content:

zen = 'xxxxxx'
testdata = [zen.split('\n') * 3] * 10

fafhrd91 · 2017-08-02T02:46:16Z

Hmm, that's strange. I am afraid my fix fixes side effects and not actual problem.

messense · 2018-01-09T12:32:30Z

This issue still occurs sometimes with Vec<Vec<String>> as argument. The cause may be nested iterators.

fafhrd91 · 2018-01-17T16:15:06Z

could you create repo with code with reproducible bug

messense · 2018-01-20T11:59:02Z

I have created a repo for reproducing the issue here: https://github.com/messense/pyo3-issue-71

konstin · 2018-11-24T12:22:33Z

Should also be fixed by 0.5.1

messense mentioned this issue Jul 31, 2017

Fix PyIterator segfault #72

Closed

fafhrd91 added a commit that referenced this issue Aug 1, 2017

temp fix for #71

5dad22e

messense closed this as completed Aug 2, 2017

messense reopened this Jan 9, 2018

messense added the bug label Jan 9, 2018

Shnatsel mentioned this issue Jul 19, 2018

Run fuzzing tools to discover safety holes mre/hyperjson#30

Open

gsolberg mentioned this issue Jul 26, 2018

Segfault with vector of vectors. #193

Closed

konstin closed this as completed Nov 24, 2018

birkenfeld mentioned this issue May 26, 2019

SIGSEGV when PyIterator is given an integer #494

Closed

kngwyu mentioned this issue Jul 18, 2019

Handle null in PyIterator::from_object to fix #494 #544

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

segfault with large data #71

segfault with large data #71

messense commented Jul 31, 2017

messense commented Jul 31, 2017

fafhrd91 commented Jul 31, 2017

messense commented Jul 31, 2017

messense commented Jul 31, 2017

fafhrd91 commented Jul 31, 2017

messense commented Jul 31, 2017

fafhrd91 commented Jul 31, 2017 via email

messense commented Jul 31, 2017

messense commented Jul 31, 2017

fafhrd91 commented Jul 31, 2017

messense commented Jul 31, 2017

fafhrd91 commented Jul 31, 2017

messense commented Jul 31, 2017

fafhrd91 commented Jul 31, 2017

messense commented Jul 31, 2017

messense commented Jul 31, 2017 •

edited

Loading

fafhrd91 commented Aug 1, 2017

messense commented Aug 2, 2017

fafhrd91 commented Aug 2, 2017

messense commented Aug 2, 2017

messense commented Aug 2, 2017

fafhrd91 commented Aug 2, 2017 •

edited

Loading

messense commented Jan 9, 2018 •

edited

Loading

fafhrd91 commented Jan 17, 2018

messense commented Jan 20, 2018

konstin commented Nov 24, 2018

segfault with large data #71

segfault with large data #71

Comments

messense commented Jul 31, 2017

messense commented Jul 31, 2017

fafhrd91 commented Jul 31, 2017

messense commented Jul 31, 2017

messense commented Jul 31, 2017

fafhrd91 commented Jul 31, 2017

messense commented Jul 31, 2017

fafhrd91 commented Jul 31, 2017 via email

messense commented Jul 31, 2017

messense commented Jul 31, 2017

fafhrd91 commented Jul 31, 2017

messense commented Jul 31, 2017

fafhrd91 commented Jul 31, 2017

messense commented Jul 31, 2017

fafhrd91 commented Jul 31, 2017

messense commented Jul 31, 2017

messense commented Jul 31, 2017 • edited Loading

fafhrd91 commented Aug 1, 2017

messense commented Aug 2, 2017

fafhrd91 commented Aug 2, 2017

messense commented Aug 2, 2017

messense commented Aug 2, 2017

fafhrd91 commented Aug 2, 2017 • edited Loading

messense commented Jan 9, 2018 • edited Loading

fafhrd91 commented Jan 17, 2018

messense commented Jan 20, 2018

konstin commented Nov 24, 2018

messense commented Jul 31, 2017 •

edited

Loading

fafhrd91 commented Aug 2, 2017 •

edited

Loading

messense commented Jan 9, 2018 •

edited

Loading