Test failure when Data/Sample is not available

[havardAasen]

This relates to stable/0.108 branch, tested with CoinUtils 2.11.10 and as the title says, without the directory Data.

We begin with a segfault in

Osi/Osi/src/Osi/OsiColCut.hpp

Line 248 in f705b23

bool OsiColCut::feasible(const OsiSolverInterface &im) const

This is fixed with commit 5c401b5, I'm not sure if it is a underlying problem that's still unresolved by this patch, it's obvious that the tests will fail, though I haven't looked at why column information is not available.

Whit the patch applied, the unit tests completes with 12 unexpected error

Log from unit tests

Testing OsiRowCut with OsiTestSolverInterface
Coin6001E Unable to open mps input file ../../Data/Sample/exmip1.mps
*** osirowcutSolverInterface testing issue: read MPS failed: imP->readMps(fn.c_str(), "mps") == 0
Coin6001E Unable to open mps input file ../../Data/Sample/exmip1.mps
*** osirowcutSolverInterface testing issue: read MPS failed: imP->readMps(fn.c_str(), "mps") == 0
*** osirowcutSolverInterface testing issue: consistent(IntegerModel) failed: cut.consistent(*imP)
Coin6001E Unable to open mps input file ../../Data/Sample/exmip1.mps
*** osirowcutSolverInterface testing issue: read MPS failed: imP->readMps(fn.c_str(), "mps") == 0
Testing OsiColCut with OsiTestSolverInterface
Coin6001E Unable to open mps input file ../../Data/Sample/exmip1.mps
Coin6001E Unable to open mps input file ../../Data/Sample/exmip1.mps
*** osicolcutSolverInterface testing issue: consistent(IntegerModel) failed: cut.consistent(*imP)
*** osicolcutSolverInterface testing issue: consistent(IntegerModel) failed: cut.consistent(*imP)
*** osicolcutSolverInterface testing issue: consistent(IntegerModel) failed: cut.consistent(*imP)
*** osicolcutSolverInterface testing issue: infeasible(IntegerModel) failed: cut.infeasible(*imP)
*** osicolcutSolverInterface testing issue: consistent(IntegerModel) failed: cut.consistent(*imP)
Coin6001E Unable to open mps input file ../../Data/Sample/exmip1.mps
*** osicolcutSolverInterface testing issue: infeasible(IntegerModel) failed: cut.infeasible(*imP)
*** osicolcutSolverInterface testing issue: infeasible(IntegerModel) failed: cut.infeasible(*imP)
Testing OsiCuts
Testing OsiTestSolverInterface
vol: running common unit tests.
Testing empty solver interface ... 
Coin6001E Unable to open mps input file ../../Data/Sample/exmip1.mps
*** volSolverInterface testing issue: read MPS file failed: exmip1Si->readMps(fn.c_str(), "mps") == 0
***Skipped Testing of OsiSolverInterface on Netlib problems***
***use -testOsiSolverInterface to run them.***
ERROR     osirowcut read MPS
                    imP->readMps(fn.c_str(), "mps") == 0
                    OsiRowCutTest.cpp:255
ERROR     osirowcut read MPS
                    imP->readMps(fn.c_str(), "mps") == 0
                    OsiRowCutTest.cpp:297
ERROR     osirowcut consistent(IntegerModel)
                    cut.consistent(*imP)
                    OsiRowCutTest.cpp:308
ERROR     osirowcut read MPS
                    imP->readMps(fn.c_str(), "mps") == 0
                    OsiRowCutTest.cpp:321
ERROR     osicolcut consistent(IntegerModel)
                    cut.consistent(*imP)
                    OsiColCutTest.cpp:227
ERROR     osicolcut consistent(IntegerModel)
                    cut.consistent(*imP)
                    OsiColCutTest.cpp:232
ERROR     osicolcut consistent(IntegerModel)
                    cut.consistent(*imP)
                    OsiColCutTest.cpp:236
ERROR     osicolcut infeasible(IntegerModel)
                    cut.infeasible(*imP)
                    OsiColCutTest.cpp:237
ERROR     osicolcut consistent(IntegerModel)
                    cut.consistent(*imP)
                    OsiColCutTest.cpp:242
ERROR     osicolcut infeasible(IntegerModel)
                    cut.infeasible(*imP)
                    OsiColCutTest.cpp:258
ERROR     osicolcut infeasible(IntegerModel)
                    cut.infeasible(*imP)
                    OsiColCutTest.cpp:263
ERROR     vol       read MPS file
                    exmip1Si->readMps(fn.c_str(), "mps") == 0
                    OsiSolverInterfaceTest.cpp:3905
Severity NOTE      :    0  thereof expected:    0
Severity PASSED    :  345  thereof expected:    0
Severity WARNING   :    0  thereof expected:    0
Severity ERROR     :   12  thereof expected:    0
Tests completed with 12 unexpected errors

As can be seen from the log, 4 tests fails because it can't read ../../Data/Sample/exmip1.mps, 3 tests fails with the patch I mentioned earlier, column information is not available.

I haven't looked at why the last 5 tests fails, however, of these 5 failed tests, only 1 still fails when Data/Sample is available.

---

If I run the tests when Data/Sample/exmip1.mps is available I get 1 error.

ERROR     vol       cloning of application data
                    *(static_cast< int * >(si2->getApplicationData())) == ad
                    OsiSolverInterfaceTest.cpp:4041

This error was fixed by applying this commit 192b006

[jpfeuffer]

@tkralphs This sounds exactly like the same problem we are facing for conda
conda-forge/coin-or-osi-feedstock#15

What should we do here?

[mr-c]

We also had this issue for the Debian packaging of coinor-symphony; it would be really nice to get that sample data back

[havardAasen]

We also had this issue for the Debian packaging of coinor-symphony; it would be really nice to get that sample data back

The files exists in a different repository, the problem is licensing, or rather the lack of a license.

If the files was licensed, it wouldn't have been difficult to add them as a separate package and include it in Debian.

[jpfeuffer]

Which other repository?
And what do you suggest for conda? I mean we don't need to ship the test files but it would be great to actually be able to test things.
Can this specific test be disabled?

[havardAasen]

coin-or-tools/Data-Sample. I do not have a suggestion for Conda, as these files does not have any license. Not sure how Conda, or any other projects, handles licensing issues.

I'm not proud of the way we solved this, but it works

diff --git a/Osi/test/unitTest.cpp b/Osi/test/unitTest.cpp
index 6e486ff..0536ff1 100644
--- a/Osi/test/unitTest.cpp
+++ b/Osi/test/unitTest.cpp
@@ -31,7 +31,9 @@ using namespace OsiUnitTest;
   set, is set up for exactly this sort of comparison.  To take advantage of
   it, you'll need to edit this file and Makefile in order to get it to work.
 */
+#ifdef COIN_HAS_SAMPLE
 #define USETESTSOLVER
+#endif
 
 /*
   Some convenient undef's, to make it easy to isolate a particular solver.

[tkralphs]

Sorry for not being more on top of this. As @havar@havardAasen said, the test instances are here: https://github.com/coin-or-tools/Data-Sample. There is nothing particularly special about these instances. They were not really chosen with any purpose in mind other than just being instances that were small and had been used a lot before.

The purpose of Sample Data instance repo was for it to contain just a few instances of each different type to do some basic installation testing across the whole suite of COIN-OR solvers, nothing rigorous in terms of testing functionality or correctness. We originally had multiple different larger sets of instances of different types and it was kind of a mess, requiring downloading large amounts of data, even if all that was needed was a simple install test. The fact that it's meant to work across different project is why it's separate. This solution is lightweight and works perfectly for what we need in most cases...except for this license issue.

I guess it would be possible to find instances with a clear license that could be used instead (in the MIPLIB, for example), but it would still be some work. We could possibly add a few instances directly in the repo for some basic testing. The rationale for having a separate repo was that it contained test instances that were used across many different projects.

First, though, I believe there was already a bug fix in one of the projects for exactly this problem---the unit tests crashing when the sample data is not present. Maybe it didn't make it out into release yet? Or it was another project? I can try to look into it a bit. The unit testing should still do quite a bit, even if the sample instances are not there. It should be enough to certify that everything is working properly. So before trying to find some properly licensed instances, we should be able to fix this issue and maybe that's sufficient.