Test failures on s390x architecture

[nickva]

Running CouchDb 3.x on s390x shows a bunch of test failures seemingly related the compiling javascript code with libmozjs-60-0:s390x package.

$ docker run -it apache/couchdbci-debian:s390x-buster-erlang-20.3.8.26-1
$ cd /home/jenkins
$ git clone https://github.com/apache/couchdb.git
$ ./configure --spidermonkey-version 60 --dev
$ make
$ make eunit apps=couch_mrview suites=couch_mrview_all_docs_tests

Here is an example of a test failure and how to run it:

==> couch_mrview (eunit)
======================== EUnit ========================
module 'couch_mrview_all_docs_tests'
  _all_docs view tests
    couch_mrview_all_docs_tests:70: should_query...*failed*
in function couch_mrview_all_docs_tests:'-should_query/1-fun-0-'/2 (test/eunit/couch_mrview_all_docs_tests.erl, line 70)
**error:{assertEqual,[{module,couch_mrview_all_docs_tests},
              {line,70},
              {expression,"Result"},
              {expected,{ok,[{meta,[{total,11},{offset,0}]},
                             {row,[{id,<<"1">>},{key,<<"1">>},{value,{...}}]},
                             {row,[{id,<<"10">>},{key,<<...>>},{value,...}]},
                             {row,[{id,<<...>>},{key,...},{...}]},
                             {row,[{id,...},{...}|...]},
                             {row,[{...}|...]},
                             {row,[...]},
                             {row,...},
                             {...}|...]}},
              {value,{ok,[{meta,[{total,10},{offset,0}]},
                          {row,[{id,<<"1">>},{key,<<...>>},{value,...}]},
                          {row,[{id,<<...>>},{key,...},{...}]},
                          {row,[{id,...},{...}|...]},
                          {row,[{...}|...]},
                          {row,[...]},
                          {row,...},
                          {...}|...]}}]}
  output:<<"">>

    couch_mrview_all_docs_tests:80: should_query_with_range...*failed*
in function couch_mrview_all_docs_tests:'-should_query_with_range/1-fun-0-'/2 (test/eunit/couch_mrview_all_docs_tests.erl, line 80)
**error:{assertEqual,[{module,couch_mrview_all_docs_tests},
              {line,80},
              {expression,"Result"},
              {expected,{ok,[{meta,[{total,11},{offset,3}]},
                             {row,[{id,<<"3">>},{key,<<"3">>},{value,{...}}]},
                             {row,[{id,<<"4">>},{key,<<...>>},{value,...}]},
                             {row,[{id,<<...>>},{key,...},{...}]}]}},
              {value,{ok,[{meta,[{total,10},{offset,3}]},
                          {row,[{id,<<"3">>},{key,<<...>>},{value,...}]},
                          {row,[{id,<<...>>},{key,...},{...}]},
                          {row,[{id,...},{...}|...]}]}}]}
  output:<<"">>
...

  [done in 0.336 s]
[done in 1.609 s]
=======================================================
  Failed: 5.  Skipped: 0.  Passed: 0.

The logs are full of these errors:

**throw:{compilation_error,<<"(new SyntaxError(\"missing } in compound statement\", \"<unknown>\", 3))"...>>}
  output:<<"">>

[Nayana-ibm]

Hi @nickva
I tried executing same steps mentioned by you however make itself failing at my end.

$ make
==> couch_epi (compile)
ERROR: OTP release 20 does not match required regex 21|22|23|24
ERROR: compile failed while processing /home/jenkins/couchdb/src/couch_epi: rebar_abort

Could you please share

1. output of configure and make command
2. if any other dependency/configuration is used

[nickva]

@Nayana-ibm

Thanks for taking a look. Sorry I forgot to mention that this is for the 3.x branch only. So before the configure step, we'd switch to the 3.x branch.

git checkout 3.x && ./configure --dev  --spidermonkey-version 60
...

FoundationDB which is needed for the main branch (the future 4.x) can't compile on either ppc64le or s390x yet.

[Nayana-ibm]

Few observations:
Checked apache/couchdbci-debian:buster-erlang-20.3.8.26-1 for amd64 and could see package couch-libmozjs185-dev installed
However on s390x image, couldn't find this package.

Tried building and installing erlang 22.2, elixir 1.10.2, node v14.2.0 and spider monkey js-1.8.5 on ub18.04. Test are passing with couchdb 3.x with this.

[nickva]

Good observation @Nayana-ibm. We use libmozjs185 on ppc64le so perhaps there some issue with the newer JS library and we should switch to libmozjs185 for s390x for the time being.

[SriramaSharma]

@nickva So what is the plan you suggest to proceed further on this ?
As @Nayana-ibm has pointed, couch-libmozjs185-dev which is installed on apache/couchdbci-debian:buster-erlang-20.3.8.26-1 but not available on apache/couchdbci-debian:s390x-buster-erlang-20.3.8.26-1.

Can we build the js185 on s390x using the steps specified here - https://github.com/linux-on-ibm-z/scripts/blob/master/CouchDB/3.1.1/build_couchdb.sh#L170 ?
OR
Should we have s390x version of couch-libmozjs185-dev package built similar to how its done on amd64 ?

[nickva]

Ideally we'd find why mozjs60 (spidermonkey version 60) doesn't work on s390x but for the time being we should probably try to build as a deb/rpm package just like we build it for amd64 and ppc64le.

The best place to start is with the readme file in https://github.com/apache/couchdb-pkg

Then https://github.com/apache/couchdb-pkg/search?q=ppc64le shows how ppc64le is handled, I imagine s390x would look similar.

[nickva]

Sorry to bother @Nayana-ibm, not sure if you'd even know about it, but it looks like the s390x Jenkins worker donated by IBM (couchdb01) is down.

To help identify it, the IP can be seen in: https://github.com/apache/couchdb-infra-cm/blob/02a0d1ed2dbbebf2b88d5fcf4e94d9022acd2362/production#L167-L179

Did it accidentally got shut down or removed and just needs a restart or is it gone for good?

[nickva]

s390x node was removed from the CI workers lists for now apache/couchdb-infra-cm@67a6468

[nickva]

With the later OS variants and a new KVM based s390x CI instances all tests now pass the s390x is part of the regular full CI run.