Improper handling of query parameters depending on their order for relational arrays.

[przemyslaw-szurmak]

During investigation of iOS SDK bug parse-community/Parse-SDK-iOS-OSX#1067 I've noticed that one of my queries fails when working on 32 bit device. At first, I thought it's due to recent SDK update from 1.13.0 to 1.14.2 but I was able to reproduce it with both SDK versions. After few hours of debugging how requests are made on 32/64 bit SDK I came to conclusion that ParseServer treats differently requests that queries fields with one direct object pointer vs array of those pointers. Because of the way how iOS SDK stores request parameters internally (in NSDictionary which is unordered collection) JSONs producend on 32 bit devices are different then the one from 64 bit devices (presented below) and requests on 32 bit devices is failing.

Steps to reproduce

1. Ids known for two objects with type fooo

    let fooId = "SLa1pur9QB"
    let fooId2 = "zDtZFtdx9X"

2. Couple of objects("GameScore") with relation array to "fooo" objects

let foo = PFObject(withoutDataWithClassName: "fooo", objectId: fooId)
let foo2 = PFObject(withoutDataWithClassName: "fooo", objectId: fooId2)
let gameScore = PFObject(className:"GameScore")
gameScore["foooVal"] = [foo, foo2]

3. Query to find objects of type GameScore where object foo2 is stored in "fooVal" array

let foo = PFObject(withoutDataWithClassName: "fooo", objectId: fooId)
let gameScoreQuery = PFQuery(className:"GameScore")
gameScoreQuery.whereKey("foooVal", equalTo: foo)
gameScoreQuery.findObjectsInBackground { (objects, error) in
    print("error \(error)")
    print("objects = \(objects?.count)")
}

I'll also present case where this doesn't matter due to fact that fields containg just one relation objects are stored differently by ParseServer

1. Save object of type GameScore2 with direct relation to fooo

let foo2 = PFObject(withoutDataWithClassName: "fooo", objectId: fooId2)
let gameScore = PFObject(className:"GameScore2")
gameScore["foooVal"] = foo2

2. Query

let foo2 = PFObject(withoutDataWithClassName: "fooo", objectId: fooId2)
let gameScoreQuery = PFQuery(className:"GameScore2")
gameScoreQuery.whereKey("foooVal", equalTo: foo2)
gameScoreQuery.findObjectsInBackground { (objects, error) in
    print("error \(error)")
    print("objects = \(objects?.count)")
}

Expected Results

Ordering of parameters in both queries should not influence results.

Actual Outcome

1. Query on array field

Produced JSON on 32 bit devices:

{"where":{"foooVal":{"objectId":"SLa1pur9QB","className":"fooo","__type":"Pointer"}},"_method":"GET"}

Produced JSON on 64 bit devices:

{"where":{"foooVal":{"__type":"Pointer","className":"fooo","objectId":"SLa1pur9QB"}},"_method":"GET"}

Result:
On 64 bit devices all values are returned correctly, on 32 bit empty array is returned(refer to logs section)

2. Query on direct releation field

32 bit device:

{"where":{"foooVal":{"objectId":"zDtZFtdx9X","className":"fooo","__type":"Pointer"}},"_method":"GET"}

64 bit device

{"where":{"foooVal":{"__type":"Pointer","className":"fooo","objectId":"zDtZFtdx9X"}},"_method":"GET"}

Result:
In both cases proper values is returned

Environment Setup

• Server

  • parse-server version: 2.2.25-beta.1 commit: c8823f2
  • Operating System: 64bit Amazon Linux 2016.09 v3.1.0 running Node.js
  • Hardware: t2.micro instance
  • Localhost or remote server?: AWS Cloud

• Database

  • MongoDB version: 3.2.11
  • Hardware: mLab Sandbox
  • Localhost or remote server?: mLab

Logs/Trace

1. query on relation array
64 bit request/response log from server:

verbose: REQUEST for [GET] /parse/classes/GameScore: {
  "where": {
    "foooVal": {
      "__type": "Pointer",
      "className": "fooo",
      "objectId": "SLa1pur9QB"
    }
  }
}

verbose: RESPONSE from [GET] /parse/classes/GameScore: {
  "response": {
    "results": [
      SOME_PROPER_VALUES
    ]
  }
}

32 bit request/response log from server:

verbose: REQUEST for [GET] /parse/classes/GameScore: {
  "where": {
    "foooVal": {
      "objectId": "SLa1pur9QB",
      "className": "fooo",
      "__type": "Pointer"
    }
  }
}

verbose: RESPONSE from [GET] /parse/classes/GameScore: {
  "response": {
    "results": []
  }
} results=[]

2. query on direct relation field
64 bit request/response log from server

verbose: REQUEST for [GET] /parse/classes/GameScore2: {
  "where": {
    "foooVal": {
      "__type": "Pointer",
      "className": "fooo",
      "objectId": "zDtZFtdx9X"
    }
  }
}
verbose: RESPONSE from [GET] /parse/classes/GameScore2: {
  "response": {
    "results": [
      SOME_PROPER_VALUES
    ]
  }
}

32 bit request/response log from server:

verbose: REQUEST for [GET] /parse/classes/GameScore2: {
  "where": {
    "foooVal": {
      "objectId": "zDtZFtdx9X",
      "className": "fooo",
      "__type": "Pointer"
    }
  }
}
verbose: RESPONSE from [GET] /parse/classes/GameScore2: {
  "response": {
    "results": [
    SOME_PROPER_VALUES
    ]
  }
}

I guess that if field contains direct relation to object query is translated from above form to CLASS_NAME$OBJECT_ID before sending it to MongoDB, and in case of field containg array query from above is passed directly to MongoDB which is why it's returning 0 objects. So either such translation should happen in both cases or SDK should create queries in specific order (although I coulnd't find any statement in documentation for REST Api about such requirement) or maybe there's a way to convince MongoDB to not look at parameter order, just "content".

[mkoslacz]

I encounter the same problem. Any updates on that?

[lenart]

I'm trying to understand why & where this occurs. It's a bit early but it looks like relying on indexes somewhere here in MongoTransform.js could be the reason.

Is anyone currently on this issue?

[flovilmart]

I'll try to have a look before end of week

[przemyslaw-szurmak]

@lenart @flovilmart thanks for looking at the issue. I was loosing hope already ;)

I will take a look at what @lenart suggested.

[flovilmart]

I haven't had time to give it much toughts last week. Any progress on investigating? That should be quite trivial to solve.

[lenart]

Thanks for follow up @flovilmart. Unfortunately I haven't able to get the resources needed to implement a proper fix for this issue. We've decided to workaround using a middleware that reorders the input params for Pointer.

For the impatient ones here's the code

// /index.js
var app = express();
...
app.use(bodyParser.json());
var PointerParamsFix = require('./lib/pointer-params-fix');
app.use(PointerParamsFix);
...

and the middleware

// /lib/pointer-params-fix.js

// Middleware for reordering params for Pointer objects
// -----------------------------------------------------
// Without this middleware parse-server responds with an empty array each time
// Pointer properties are not in this order: __type, className, objectId.
// This script checks the params and reorders any Pointer objects found
//
// Github Issue: https://github.com/ParsePlatform/parse-server/issues/3169
var PointerParamsFix = function(req, res, next) {
  var where = req.body.where
  if (where) {
    for (var key in where) {
      if (where[key].hasOwnProperty('__type')) {
        if (where[key].__type === 'Pointer') {
          where[key] = {
            "__type": where[key].__type,
            "className": where[key].className,
            "objectId": where[key].objectId,
          }
        }
      }
    }
  }
  next();
};

exports = module.exports = PointerParamsFix;

I know this requires to inject this middleware each time parse-server module is updated but it's the best we can currently afford.

[erikdrobne]

I set up parse server, did the debugging and spotted the consequent issue.

If query parameters inside where statement are in incorrect order (e.g. className after objectId),

{"where":{"foo": {"__type":"Pointer", "objectId":"a1b2c3d4e5", "className":"bar"}}}

then method named mongoObjectToParseObject doesn't get executed, because mongoObjectparameter is empty.

https://github.com/ParsePlatform/parse-server/blob/master/src/Adapters/Storage/Mongo/MongoTransform.js#L737

This behaviour is caused by find() method in MongoStorageAdapter.js. This method is using lower level collection.find() to retrieve data from the database based on query parameters. It produces promise which is in case of correct parameters order resolved into object collection, otherwise it's resolved empty [].

https://github.com/ParsePlatform/parse-server/blob/master/src/Adapters/Storage/Mongo/MongoStorageAdapter.js#L328

https://github.com/ParsePlatform/parse-server/blob/master/src/Adapters/Storage/Mongo/MongoStorageAdapter.js#L131

https://github.com/ParsePlatform/parse-server/blob/master/src/Adapters/Storage/Mongo/MongoCollection.js#L16

Modifying collection.find() would be difficult to test and to ensure correctness.
I don't know parse server arhitecture well, but would anyway suggest to handle the parameters order somewhere inside MongoStorageAdapter file. I think that something similar that @lenart wrote would be a good starting point.

[laptobbe]

@lenart Thanks for your solution. I found that it did not cover all the cases for nested queries, such as $or or matching queries. Based on your solution I updated it to iterate the entire where object and update all pointers recursively.

// /lib/pointer-params-fix.js

// Middleware for reordering params for Pointer objects
// -----------------------------------------------------
// Without this middleware parse-server responds with an empty array each time
// Pointer properties are not in this order: __type, className, objectId.
// This script checks the params and reorders any Pointer objects found
//
// Github Issue: https://github.com/ParsePlatform/parse-server/issues/3169
var PointerParamsFix = function(req, res, next) {
  var where = req.body.where
  RecusivePointerParamsFix(where);
  next();
};

var RecusivePointerParamsFix = function(object) {
  if (object) {
    for (var key in object) {
      if (object[key].hasOwnProperty('__type')) {
        if (object[key].__type === 'Pointer') {
          object[key] = {
            "__type": object[key].__type,
            "className": object[key].className,
            "objectId": object[key].objectId,
          }
        }
      } else if (Array.isArray(object[key])){
        for (var subKey in object[key]) {
          RecusivePointerParamsFix(object[key][subKey]);
        }
      } else if (object[key] != null && typeof object[key] == 'object') {
        RecusivePointerParamsFix(object[key]);
      }
    }
  }
};

exports = module.exports = PointerParamsFix;

[flovilmart]

I completely dropped the ball on that one, need to look it up a bit more.