Build Crash

[lavahot]

I have no idea why this is crashing. Calls to arc later in the log work just fine seemingly.

[android_app] $ arc call-conduit differential.querydiffs ********
FATAL: net.sf.json.JSONArray cannot be cast to net.sf.json.JSONObject
java.lang.ClassCastException: net.sf.json.JSONArray cannot be cast to net.sf.json.JSONObject
at com.uber.jenkins.phabricator.conduit.Differential.(Differential.java:52)
at com.uber.jenkins.phabricator.PhabricatorBuildWrapper.setUp(PhabricatorBuildWrapper.java:91)
at hudson.model.Build$BuildExecution.doRun(Build.java:156)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537)
at hudson.model.Run.execute(Run.java:1744)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:374)

[ascandella]

What is your DIFF_ID parameter set to?

[ascandella]

so the line that's crashing on is:

        this.rawJSON = (JSONObject) ((JSONObject) query.get("response")).get(diffID);

it sounds like your response is an array, when it should be an object. we definitely need better error handling here, but perhaps your arc is out of date? what does arc version tell you on your jenkins slaves?

when you run this command, what does the response object look like?

echo '{"ids": [123]}' | arc call-conduit differential.querydiffs | python -mjson.tool

for me, it starts with:

{
    "error": null,
    "errorMessage": null,
    "response": {
        "123": {

[lavahot]

My DIFF_ID is "6" (without quotes). On my slave I upgrade arc every time the slave comes online, which is no more than every time the job builds, but on average is less frequent.

For reference, here is the output of arc --version:

arcanist 7d15b85a1bc099e34a3bbf194fc8f92c33bc69a9 (8 Jun 2015)
libphutil 92882eb9404da16ba2ea498c1182cd3c1278877f (8 Jun 2015)

The output of echo '{"ids": [123]}' | arc call-conduit differential.querydiffs | python -mjson.tool is:

{
"error": null,
"errorMessage": null,
"response": []
}

[ascandella]

hmm... what about a valid diff ID? (1)

[ascandella]

also, what version/sha of phabricator backend are you running? seems like it should be returning an empty object, not an empty array

[ascandella]

err what is the output of this:

echo '{"ids": [6]}' | arc call-conduit differential.querydiffs | python -mjson.tool

[lavahot]

OOOOOHHHHHH. Yes, the output for that is similar to what you mentioned earlier.

{
"error": null,
"errorMessage": null,
"response": {
"6": {

I just now updated my Phabricator instance and reran the build. It's still failing. Hmm...

[ascandella]

Still failing with the same error?

[lavahot]

Yes, identical.

[ascandella]

I'm unclear on how that would occur. You're passing DIFF_ID=6 to the build? Either way, if you want to review the associated PR, I can deploy that and we can get some more debugging output.

[lavahot]

Yes, that's correct. It occurs to me that something else may be going on because Jenkins is now posting as me and not the jenkins bot in the related Diff. Let me see if my arcrc is correct before we commit to a PR.

[lavahot]

Yeah, looks like the arcrc didn't matter. I'll review the PR.

[lavahot]

I thought I would dump my whole (censored) log here for more context. Seems like there's more here.

Started by remote host 191.236.121.231
Building remotely on AzureNUMBER (Linux) in workspace /home/user/workspace/android_app

git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url [email protected]:Company/Android_Project.git # timeout=10
Fetching upstream changes from [email protected]:Company/Android_Project.git
git --version # timeout=10
using GIT_SSH to set credentials A new key for jenkins to connect to my github account
git -c core.askpass=true fetch --tags --progress [email protected]:Company/Android_Project.git +refs/heads/:refs/remotes/origin/
git rev-parse refs/remotes/origin/master^{commit} # timeout=10
git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 46bdc7e526d917135644142760b19bc78bbfda9c (refs/remotes/origin/master)
git config core.sparsecheckout # timeout=10
git checkout -f 46bdc7e526d917135644142760b19bc78bbfda9c
git rev-list 46bdc7e526d917135644142760b19bc78bbfda9c # timeout=10
[android_app] $ arc call-conduit differential.querydiffs ********
FATAL: net.sf.json.JSONArray cannot be cast to net.sf.json.JSONObject
java.lang.ClassCastException: net.sf.json.JSONArray cannot be cast to net.sf.json.JSONObject
at com.uber.jenkins.phabricator.conduit.Differential.(Differential.java:52)
at com.uber.jenkins.phabricator.PhabricatorBuildWrapper.setUp(PhabricatorBuildWrapper.java:91)
at hudson.model.Build$BuildExecution.doRun(Build.java:156)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537)
at hudson.model.Run.execute(Run.java:1744)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:374)
MicrosoftAzureStorage - Build failed, hence canceling the upload of the artifacts to Azure storage
[android_app] $ arc call-conduit differential.querydiffs
Usage Exception: YOU NEED TO AUTHENTICATE TO CONTINUE

You are trying to connect to a server (companyphab.cloudapp.net) that you do not have any credentials stored for.

To retrieve and store credentials for this server, run this command:

  $ arc install-certificate

[arcanist] returned non-zero exit code 1
[arcanist] output:
[arcanist] unable to fetch differential
Finished: FAILURE

Actually it kind of looks like the token might not be working, but I've double checked that and it is active in Phabricator and entered correctly in the Jenkins Settings.

[ascandella]

ok, just published 1.5, try upgrading and seeing what the output arc is getting when it's choking...

[lavahot]

Hey @sectioneight, any idea when 1.5 will post? Didn't Jenkins say about 4 hours was normal?

[ascandella]

It's available on maven:
http://repo.jenkins-ci.org/public/org/jenkins-ci/plugins/phabricator-plugin/

Have you tried hitting the "update" button from your jenkins plugin manager
to refresh manually?

On Tue, Jun 9, 2015 at 8:43 PM, Taylor Mansfield [email protected]
wrote:

Hey @sectioneight https://github.com/sectioneight, any idea when 1.5
will post? Didn't Jenins say about 4 hours was normal?

—
Reply to this email directly or view it on GitHub
#30 (comment)
.

[lavahot]

Hey, it finally showed up! Bad news though:

[android_app] $ arc call-conduit differential.querydiffs ********
[arcanist] unable to apply patch
No 'response' object found in conduit call: (JSONObject["response"] is not a JSONObject.) {
"error": null,
"errorMessage": null,
"response": []
}

[lavahot]

I'm starting to suspect it has something to do with the callback:

[android_app] $ arc call-conduit differential.querydiffs
Usage Exception: YOU NEED TO AUTHENTICATE TO CONTINUE

You are trying to connect to a server (nvdphab.cloudapp.net) that you do not have any credentials >stored for.

To retrieve and store credentials for this server, run this command:

 $ arc install-certificate

[ascandella]

It sounds like there are two separate issues going on here. The first is that your build is getting kicked off without a DIFF_ID (or that's not being properly sent from harbormaster). Have you verified that the build parameters are being set properly when triggered by phabricator?

For example, on

https://jenkins.example.com/job/test-thing/1/parameters/

Since you mentioned that when you pass a valid BUILD_ID (e.g. 6) into conduit, it queries properly, it sounds like some of the ends aren't meeting in your job.

The second is the "callback" you just mentioned. At what stage in the build is that?

[lavahot]

I am 100% certain that the DIFF_ID and PHID are passed. In the parameter log for each build, I can see that they are filled with the proper parameters. I double checked the HTTP string against the other build I know works and it seems to be constructed correctly.

By "callback" I mean the post-build action that sends information back to Phabricator by commenting.

[lavahot]

It's possible that there might be some interference from some other plugin. I will systematically remove these options until there are none left (or until it works).

[lavahot]

Okay, I feel like I'm chasing my tail here. Stripping out the plugin stuff did nothing. I re-ran that string you asked me and it gave me an authentication error, then I ran it with my api token like this: echo '{"ids": [12]}' | arc call-conduit differential.querydiffs --conduit-token=api-somelongtoken | python -mjson.tool and it spat out the empty array without error instead of the contents of the diff. In fact, I ran all of the diffs I have through this and numbers associated with my good working build pulled the data and numbers associated with this build returned the empty array.

Something is fishy here. I do not think it is this plugin. It could very well be something on the phabricator end.

[lavahot]

And it's not just my slave install, same thing happens on my laptop...

[lavahot]

Wow. Okay. So I had restricted the repo to just the members of the project responsible for it, and my jenkins bot was not a member of that project. When using the api key for the jenkins bot it enforced the permissions by sending an empty array. I'm stupid. The good news is that you might be able to add an error message sending people in that direction.

[ascandella]

Oh wow, that is strange. Can you paste me the full error response? Is there
an error and or an error message? Possibly something we should fix
upstream. Glad things are finally working though!

On Wednesday, June 10, 2015, Taylor Mansfield [email protected]
wrote:

Closed #30 #30.

—
Reply to this email directly or view it on GitHub
#30 (comment)
.

[lavahot]

There is no error message. It just gives an empty array like has been previously posted here. Seems like there should be an error there explicitly and not nulls.

[ascandella]

Created https://secure.phabricator.com/T8513 to track upstream. Thanks for helping debug this.

[lavahot]

Thanks, @sectioneight!