An Ember problem causes Detailed Network test to fail on RedHat-7 VM and on MultiRank

[jpvandy]

This issue is about an Ember file naming problem that breaks Scheduler Detailed Network test
Issue 147 is about an occasional Ember assert about "bufLen" buffer usage problem.
Issue 274 is about an occasional Ember Time Limit.

The test was running on all other tested platforms. The failure appears to have to do with over writing the motif-xx.log files.
(These failures can trigger another problem, the SQE timelimit processor really attempts to deal with a time limit for an SST run. the Detailed network test has a python wrapper that loops thru multiple sst invocations. Consequently, it can fail to terminate a wrapper loop.)

The failure is also observed with MULTI_RANK execution on the Serialization Branch

[vjleung]

John,

So is it just how the redhat-7 vm handles write permissions by default.

Vitus

Sent from my iPhone

On Feb 8, 2016, at 3:36 PM, John <[email protected]mailto:[email protected]> wrote:

The test was running on all other tested platforms. The failure appears to have to do with over writing the motif-xx.log files.
(These failures can trigger another problem, the SQE timelimit processor really attempts to deal with a time limit for an SST run. the Detailed network test has a python wrapper that loops thru multiple sst invocations. Consequently, it can fail to terminate a wrapper loop.)

Reply to this email directly or view it on GitHubhttps://github.com//issues/108.

[nmhamster]

I think this is really down to how SST Ember actually does file opening. There is a potential ordering effect here on who owns the final FILE*. Its more of a bug in Ember which is now getting exposed.

[nmhamster]

John, can you try running this again and see if it restores old behavior at least? There is still an issue where a "job" spans multiple SST ranks.

[jpvandy]

Nightly seems to have restored the old behavior.

I’m VPN from home at the moment. Be in after a bit.
Was just trying to run on my sandbox of yesterday, when your email came in

John

From: Si Hammond [mailto:[email protected]]
Sent: Tuesday, February 09, 2016 8:06 AM
To: sstsimulator/sst-elements
Cc: Vandyke, John P
Subject: [EXTERNAL] Re: [sst-elements] The scheduler Detailed Network test fails on RedHat-7 VM (#108)

John, can you try running this again and see if it restores old behavior at least? There is still an issue where a "job" spans multiple SST ranks.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/108#issuecomment-181906422.

[nmhamster]

John,

Can you confirm the test passes on RedHat 7.X?

Si Hammond

Scalable Computer Architectures
Sandia National Laboratories, NM, USA

On 2/9/16, 8:09 AM, "John" [email protected] wrote:

Nightly seems to have restored the old behavior.

I’m VPN from home at the moment. Be in after a bit.
Was just trying to run on my sandbox of yesterday, when your email came
in

John

From: Si Hammond [mailto:[email protected]]
Sent: Tuesday, February 09, 2016 8:06 AM
To: sstsimulator/sst-elements
Cc: Vandyke, John P
Subject: [EXTERNAL] Re: [sst-elements] The scheduler Detailed Network
test fails on RedHat-7 VM (#108)

John, can you try running this again and see if it restores old behavior
at least? There is still an issue where a "job" spans multiple SST ranks.

—
Reply to this email directly or view it on
GitHub<#108 (comment)
t-181906422>.

—
Reply to this email directly or
view it on GitHub
<#108 (comment)
08291>.

[jpvandy]

The nightly FAILED on RedHat-7 as it had previously. It is looping on an empty motif-3.log file.
After updating ember, I get the same result when I run it on the VM.

The other platforms that run it are happy.
John
From: Si Hammond [mailto:[email protected]]
Sent: Tuesday, February 09, 2016 2:01 PM
To: sstsimulator/sst-elements
Cc: Vandyke, John P
Subject: [EXTERNAL] Re: [sst-elements] The scheduler Detailed Network test fails on RedHat-7 VM (#108)

John,

Can you confirm the test passes on RedHat 7.X?

Si Hammond

Scalable Computer Architectures
Sandia National Laboratories, NM, USA

On 2/9/16, 8:09 AM, "John" [email protected] wrote:

Nightly seems to have restored the old behavior.

I’m VPN from home at the moment. Be in after a bit.
Was just trying to run on my sandbox of yesterday, when your email came
in

John

From: Si Hammond [mailto:[email protected]]
Sent: Tuesday, February 09, 2016 8:06 AM
To: sstsimulator/sst-elements
Cc: Vandyke, John P
Subject: [EXTERNAL] Re: [sst-elements] The scheduler Detailed Network
test fails on RedHat-7 VM (#108)

John, can you try running this again and see if it restores old behavior
at least? There is still an issue where a "job" spans multiple SST ranks.

—
Reply to this email directly or view it on
GitHub<#108 (comment)
t-181906422>.

—
Reply to this email directly or
view it on GitHub
<#108 (comment)
08291>.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/108#issuecomment-182066273.

[jpvandy]

Nothing appears to have changed with respect to the file over write problem.
(The SQE problem of this particular infinite loop has been corrected in the Test Suite.)
Is this a scheduler bug or an ember bug?

[vjleung]

John,

I have lost track of this a little bit. The last thing I remember is Si saying the problem is not in the scheduler.

Vitus

Sent from my iPhone

On Feb 19, 2016, at 10:54 AM, John <[email protected]mailto:[email protected]> wrote:

Nothing appears to have changed with respect to the file over write problem.
(The SQE problem of this particular infinite loop has been corrected in the Test Suite.)
Is this a scheduler bug or an ember bug?

Reply to this email directly or view it on GitHubhttps://github.com//issues/108#issuecomment-186331237.

[nmhamster]

This is an issue with Ember. :-(

S.

Si Hammond
Scalable Computer Architectures
Center for Computing Research
Sandia National Laboratories, NM, USA
[Sent from Remote Connection, Please excuse typos]

From: vjleung <[email protected]mailto:[email protected]>
Reply-To: sstsimulator/sst-elements <[email protected]mailto:[email protected]>
Date: Friday, February 19, 2016 at 11:05 AM
To: sstsimulator/sst-elements <[email protected]mailto:[email protected]>
Cc: Simon Hammond <[email protected]mailto:[email protected]>
Subject: [EXTERNAL] Re: [sst-elements] The scheduler Detailed Network test fails on RedHat-7 VM (#108)

John,

I have lost track of this a little bit. The last thing I remember is Si saying the problem is not in the scheduler.

Vitus

Sent from my iPhone

On Feb 19, 2016, at 10:54 AM, John <[email protected]mailto:[email protected]mailto:[email protected]> wrote:

Nothing appears to have changed with respect to the file over write problem.
(The SQE problem of this particular infinite loop has been corrected in the Test Suite.)
Is this a scheduler bug or an ember bug?

Reply to this email directly or view it on GitHubhttps://github.com//issues/108#issuecomment-186331237.

Reply to this email directly or view it on GitHubhttps://github.com//issues/108#issuecomment-186334833.

[jpvandy]

At this point this is a solid failure on the RedHat-7 VM, COERHEL-7, and and on all three Multi Rank n=2 nightly test, sst-test, El Capitan, Yosemite.

[nmhamster]

@jpvandy - can you post the error into the issue please? Thank you!

[nmhamster]

Ok, no problem, I already have it! Sorry for the spam!

[jwilso]

@afrodri : Si should evaluate whether this is still a 6.0.0 issue.

[nmhamster]

Fixed.

[jpvandy]

This is not fixed, it may be deferred, or it may be "not to be fixed", but it's not fixed.

[jpvandy]

When the openmpi version was updated to 1.8.8, this problem became ubiquitous, across all platforms. It was fixed, with PR 498 on Nov. 8th.