XML to_s error

[jkthorne]

I am trying to write XMLs to a file and I am getting this error.

Invalid memory access (signal 11) at address 0x0
[0x10e782916] *Exception::CallStack::print_backtrace:(Int32 | Nil) +118
[0x10e738a0e] __crystal_sigfault_handler +318
[0x7fff203b7d7d] _sigtramp +29
[0x7fff2720f3b4] xmlFindCharEncodingHandler +176
[0x7fff272776aa] xmlNewSaveCtxt +64
[0x7fff27277889] xmlSaveToIO +31
[0x10e8bcaf6] *XML::Node#to_xml<IO::FileDescriptor>:IO::FileDescriptor +358
[0x10e8bc986] *XML::Node#to_s<IO::FileDescriptor>:Nil +6
[0x10e8058a5] *IO::FileDescriptor@IO#<<<XML::Node>:IO::FileDescriptor +37
[0x10e805867] *IO::FileDescriptor@IO#puts<XML::Node>:Nil +39
[0x10e743819] *puts<XML::Node>:Nil +57
[0x10e8f454a] *Braintree::CLI::DisputeCreateCommand::resolve<Braintree::Operations::Dispute::Sandbox::OpenDispute.class, Braintree::CLI>:NoReturn +2378
[0x10e8f3b78] *Braintree::CLI::DisputeCreateCommand::run<Braintree::CLI>:NoReturn +280
[0x10e89be72] *Braintree::CLI#run:(Bool | Nil) +5042
[0x10e89a951] *Braintree::CLI::run:(Bool | Nil) +17
[0x10e723b2f] __crystal_main +1775
[0x10e9062c6] *Crystal::main_user_code<Int32, Pointer(Pointer(UInt8))>:Nil +6
[0x10e90612c] *Crystal::main<Int32, Pointer(Pointer(UInt8))>:Int32 +44
[0x10e72f166] main +6

When I try to reduce this bug down it it starts working again. Here are the versions of XML and Crystal but are the most updated versions on homebrew.

$ crystal --version
Crystal 0.36.1 (2021-02-02)

LLVM: 11.0.1
Default target: x86_64-apple-macosx

$ brew info libxml2
libxml2: stable 2.9.10 (bottled), HEAD [keg-only]
GNOME XML library
http://xmlsoft.org/
/usr/local/Cellar/libxml2/2.9.10_2 (280 files, 10.6MB)
...

Here is a branch with the section of code that is raising the error. Also it works when parsed as HTML.

https://github.com/wontruefree/braintree/blob/bug_xml_parse/src/cli/dispute_create_command.cr#L14-L17

[asterite]

What's the text?

[asterite]

Put another way: could you provide code that I can copy/paste into a file, run it and see it break?

[jkthorne]

I have been unable to reproduce this in a smaller code sample if I read this from a file or make it a string literal in any way it seems to work and when I use a client it fails. I will try to get a smaller sample.

Here is a sample of the full XML
https://gist.github.com/wontruefree/c9d05934c80d71e2d128c5530a93bcec

[jkthorne]

This still breaks for me but now only breaks on one macbook pro and not another. I have not been able to reproduce when not making remote calls so it is hard to make a code sample for this. I am closing this to keep the issues clean. But this might be a problem introduced with 0.36.x

[jkthorne]

I was able to narrow this down without making web requests. It seems like certain ordering of xpath methods followed by a puts breaks the program. Here is a link to the working and broke code samples.

Here is the broken sample. https://gist.github.com/wontruefree/f7efd84b0cbd4eb1d2f1f32bfe249e2a#file-xml-bug

$ crystal tmp.cr
<dispute>
      <id>dpp74szzb5srtw4d</id>
      <global-id>ZGlzcHV0ZV9kcHA3NHN6emI1c3J0dzRk</global-id>
      <amount>9363.36</amount>
      <amount-disputed>9363.36</amount-disputed>
      <amount-won>0.00</amount-won>
      <case-number>CB328430743124</case-number>
      <created-at type="datetime">2021-02-05T04:40:16Z</created-at>
      <currency-iso-code>USD</currency-iso-code>
      <date-opened type="date">2021-02-05</date-opened>
      <date-won nil="true"/>
      <processor-comments nil="true"/>
      <kind>chargeback</kind>
      <merchant-account-id>kilmercorp</merchant-account-id>
      <reason>fraud</reason>
      <reason-code>62</reason-code>
      <reason-description nil="true"/>
      <received-date type="date">2021-02-05</received-date>
      <reference-number nil="true"/>
      <reply-by-date type="date">2021-02-10</reply-by-date>
      <processor-reply-by-date type="date">2021-02-14</processor-reply-by-date>
      <response-deadline type="datetime">2021-02-09T05:59:59Z</response-deadline>
      <status>open</status>
      <updated-at type="datetime">2021-02-05T04:40:16Z</updated-at>
      <original-dispute-id nil="true"/>
      <evidence type="array"/>
      <status-history type="array">
        <status-history>
          <disbursement-date type="date">2021-02-05</disbursement-date>
          <effective-date type="date">2021-02-05</effective-date>
          <status>open</status>
          <timestamp type="datetime">2021-02-05T04:40:16Z</timestamp>
        </status-history>
      </status-history>
      <transaction>
        <id>fj6df5n1</id>
        <global-id>dHJhbnNhY3Rpb25fZmo2ZGY1bjE</global-id>
        <amount>9363.36</amount>
        <created-at>2021-02-05T04:40:15Z</created-at>
        <installment-count nil="true"/>
        <order-id nil="true"/>
        <purchase-order-number nil="true"/>
        <payment-instrument-subtype>Visa</payment-instrument-subtype>
      </transaction>
    </dispute>
Invalid memory access (signal 11) at address 0x0
[0x100fa8786] *Exception::CallStack::print_backtrace:(Int32 | Nil) +118
[0x100f8494e] __crystal_sigfault_handler +318
[0x7fff203b7d7d] _sigtramp +29
[0x7fff2720f3b4] xmlFindCharEncodingHandler +176
[0x7fff272776aa] xmlNewSaveCtxt +64
[0x7fff27277889] xmlSaveToIO +31
[0x10100e366] *XML::Node#to_xml<IO::FileDescriptor>:IO::FileDescriptor +358
[0x10100e1f6] *XML::Node#to_s<IO::FileDescriptor>:Nil +6
[0x100ffe075] *IO::FileDescriptor@IO#<<<XML::Node>:IO::FileDescriptor +37
[0x100ffe037] *IO::FileDescriptor@IO#puts<XML::Node>:Nil +39
[0x100f85189] *puts<XML::Node>:Nil +57
[0x100f74f6b] __crystal_main +4427
[0x101014b06] *Crystal::main_user_code<Int32, Pointer(Pointer(UInt8))>:Nil +6
[0x10101496c] *Crystal::main<Int32, Pointer(Pointer(UInt8))>:Int32 +44
[0x100f80566] main +6

And the other one prints out he XML.

[straight-shoota]

That code runs without failure for me. Same libxml2 version, but on linux. Crystal latest is linked against LLVM 10, but master build is linked against LLVM 11.1.

$ crystal --version
Crystal 0.36.1 [c3a3c1823] (2021-02-02)

LLVM: 10.0.0
Default target: x86_64-unknown-linux-gnu

$ bin/crystal --version
Using compiled compiler at .build/crystal
Crystal 1.0.0-dev [de9351f05] (2021-02-20)

LLVM: 11.1.0
Default target: x86_64-unknown-linux-gnu

$ brew info libxml2
libxml2: stable 2.9.10 (bottled), HEAD
GNOME XML library
http://xmlsoft.org/
/home/linuxbrew/.linuxbrew/Cellar/libxml2/2.9.10_2 (281 files, 12.5MB)
...

So looks like it only fails on macos. Can anyone reproduce it?

Could also be worth trying with an older Crystal compiler and/or other libxml2 versions.

This could likely a bug in libxml2, though (or iconv or whatever would be involved).

[bcardiff]

Unable to repro osx crystal 0.36.1, llvm 11.0.1, libxml2 2.9.10

$ crystal --version
Crystal 0.36.1 (2021-02-02)

LLVM: 11.0.1
Default target: x86_64-apple-macosx

$ brew info libxml2
libxml2: stable 2.9.10 (bottled), HEAD [keg-only]
GNOME XML library
http://xmlsoft.org/
/usr/local/Cellar/libxml2/2.9.10_2 (280 files, 10.5MB)
  Poured from bottle on 2021-01-26 at 16:37:51
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/libxml2.rb
License: MIT

$ crystal run --verbose foo.ign.cr
cc _main.o D-ir.o C-rystal5858S-ystem5858D-ir.o ... stripped ... 
E-xception5858C-allS-tack5858R-epeatedF-rame.o C-rystal.o 
-o /Users/bcardiff/.cache/crystal/crystal-run-foo.ign.tmp  -rdynamic 
-L/usr/local/lib -L/usr/local/Cellar/libxml2/2.9.10_2/lib -lxml2 
-lpcre -lgc -lpthread /usr/local/Cellar/crystal/0.36.1_2/src/ext/libcrystal.a 
-L/usr/local/Cellar/libevent/2.1.12/lib -levent -liconv -ldl
<dispute>
      <id>dpp74szzb5srtw4d</id>
      <global-id>ZGlzcHV0ZV9kcHA3NHN6emI1c3J0dzRk</global-id>
      <amount>9363.36</amount>
... stripped ...
  <pin-verified type="boolean">false</pin-verified>
</transaction>

[jkthorne]

So I can reproduce this on my macbookpro(2015) but not on a macbookpro(2019).
Same versions of crystal and libxml2.

[bcardiff]

@wontruefree can you check the output of $ pkg-config libxml-2.0 --libs ?

If it is

$ pkg-config libxml-2.0 --libs
-lxml2

it means you are not using brew libxml but the default system one.

if you do export PKG_CONFIG_PATH="/usr/local/opt/libxml2/lib/pkgconfig:$PKG_CONFIG_PATH" the output should be

$ pkg-config libxml-2.0 --libs
-L/usr/local/Cellar/libxml2/2.9.10_2/lib -lxml2

With crystal run --verbose you will be able to see which link commands are used near the -lxml2

[jkthorne]

The output was:

$ pkg-config libxml-2.0 --libs
-lxml2

I used export PKG_CONFIG_PATH="/usr/local/opt/libxml2/lib/pkgconfig:$PKG_CONFIG_PATH" rebuild crystal and cleared the cache and I am still getting the same issue.
currently the pkg-config output is this.

$ pkg-config libxml-2.0 --libs
-L/usr/local/Cellar/libxml2/2.9.10_2/lib -lxml2

[beta-ziliani]

I had this same issue, and it was fixed with (Ary's suggestion):

export PATH="/opt/homebrew/opt/libxml2/bin:$PATH"
export PKG_CONFIG_PATH="${PKG_CONFIG_PATH:+$PKG_CONFIG_PATH:}/opt/homebrew/opt/libxml2/lib/pkgconfig"

(/opt/hombrew/opt/libxml2 is where I have my Brew's libxml2)

So it's a problem with certain libxml2 versions. Let's leave this open for reference, but for what it's worth it's fixed.

[Blacksmoke16]

If we so desire, we could probably fix this in the Crystal brew formula. E.g. something like what this formula is doing:

https://github.com/Homebrew/homebrew-core/blob/438d35c67eee09dec612f58eabdef0360e72a92d/Formula/p/pixz.rb#L31

Then we could always be sure pkg-config is aware of the brew implementation of it.

[HertzDevil]

I found that if we call LibXML.xmlInitParser even on non-Win32 systems then the crash goes away, at least on my M2. Can anyone else confirm this?

[Blacksmoke16]

That fixes it for me too yea. If that doesn't have any other downsides then is prob the better solution. Esp given it seems we do not actually have libxml2 as a dependency of crystal brew package anyway.

EDIT: Thinking on it more, should we make dependency more explicit. Either saying it uses the macos version, or actually add libxml2 as a dep with the pkg-config path update so Crystal will use an actual modern version of the lib...