Queries with inline data (p2p only) #88

yusefnapora · 2016-11-17T21:18:58Z

This adds a remoteQueryWithData method to the MediachainNode class, which will fetch multihash object references in query results from the remote peer and replace the multihash with {key: multihashValue, data: actualObject}

An example at the repl

א > remote.queryWithData('SELECT * FROM scratch.* LIMIT 1')
[ { simple: { stmt: [Object] } } ]
א > var res = _[0].simple.stmt
א > console.dir(res, {depth: 100})
{ id: '4XTTM6XcpMB8N8Tnkhc66DqvAotgKzRJgrZbF7SBByihtPnV8:1478207997:34006',
  publisher: '4XTTM6XcpMB8N8Tnkhc66DqvAotgKzRJgrZbF7SBByihtPnV8',
  namespace: 'scratch.moma',
  body: 
   { simple: 
      { object: 
         { key: 'QmPEeMwnRhJB1YV4sZsJV3wNtHz8jsEw1WgzjC8quqxfqU',
           data: 
            { schema: { '/': 'QmXciNhn4P6veuojsR6uRGHBu27pAztp5o1rtMkkmsTtUf' },
              data: 
               { ArtistBio: 'American, 1930–1992',
                 BeginDate: 1930,
                 ConstituentID: 1,
                 DisplayName: 'Robert Arneson',
                 EndDate: 1992,
                 Gender: 'Male',
                 Nationality: 'American',
                 ULAN: null,
                 'Wiki QID': null } } },
        refs: [ 'moma:artist:1' ],
        tags: [],
        deps: [ 'QmXciNhn4P6veuojsR6uRGHBu27pAztp5o1rtMkkmsTtUf' ] } },
  timestamp: 1478207997,
  signature: 
   Buffer [ ... ] }

TODO:

I'd like to add a new subcommand to the aleph cli that executes a query string against a remote peer and pretty prints the results, so we can get a nice one-liner to use in the Attribution Engine.

parkan · 2016-11-17T21:27:46Z

could add an invocation like aleph exec "remote.queryWithData('SELECT * FROM scratch.* LIMIT 1')"?

yusefnapora · 2016-11-17T21:36:46Z

That could work, although the repl's default printing of the query results isn't too great. For the example above you'd just get [ { simple: { stmt: [Object] } } ]

That's something we should probably tweak regardless though, to make working at the repl more useful.

…om PeerBook

… exist

parkan · 2016-11-21T17:42:00Z

src/peer/identity.js

@@ -65,12 +65,20 @@ function loadOrGenerateIdentity (filePath: string): Promise<PeerId> {
 }

 function inflateMultiaddr (multiaddrString: string): PeerInfo {
+  // total hack to patch in support for /p2p/ multiaddrs, which should
+  // be supported upstream soon


Let's actively reference this multiformats/multiaddr#27

parkan · 2016-11-21T19:29:02Z

src/peer/repl/index.js

@@ -15,16 +28,18 @@ require('yargs')
      // this is the only way to get useful config in subcommands
      'config': true,
      'global': true,
-      'default': home + 'config.json'
+      'default': path.join(home, 'config.json')


parkan · 2016-11-21T19:29:51Z

src/peer/repl/index.js

    },
    'remotePeer': {
      'global': true,
      'type': 'string',
      'describe': 'The remote peer to pair with (multiaddress)'
    }
  })
+  .coerce('config', filePath => ensureDirExists(filePath, 'config file'))


you don't want to just put this into the config key above as coerce : { filepath: ... ?

oh sure, that does sound better.

[ci skip]

parkan · 2016-11-21T19:31:12Z

src/peer/repl/util.js

+    }
+  }
+
+  return _.cloneDeepWith(result, replacer)


all the lodash pieces are broken out so you can just import lodash.clonedeepwith but it's not critical https://www.npmjs.com/package/lodash.clonedeepwith

parkan · 2016-11-21T19:32:00Z

src/peer/util.js

@@ -177,6 +179,66 @@ function promiseTimeout<T> (timeout: number, promise: Promise<T>): Promise<T> {
  })])
 }

+function flatMap<T, U> (array: Array<T>, f: (x: T) => Array<U>): Array<U> {


if only Thenables were monadic 😞

parkan · 2016-11-21T19:33:00Z

src/peer/util.js

+    try {
+      val = decode(obj.data)
+    } catch (err) {
+      val = obj.data.toString('base64')


might be nice to add a comment here explaining the logic

hmm, actually I might remove the base64 conversion here. After I wrote this, I added a pass to convert all Buffers to base64 just before printing the results. That's a better place for it than here, I think

parkan · 2016-11-21T19:33:46Z

src/peer/util.js

+    const key = _.get(value, 'object')
+    const data = objectMap[key]
+    if (data != null) {
+      return _.set(value, 'object', {key, data})


ah I guess we have all these other lodash calls too, let's take the whole package then

…nversion

yusefnapora added 13 commits November 21, 2016 10:51

add helper to extract object ids from query result

ffc3675

add remoteQueryWithData to inline query result data

cec41ef

update shrinkwrap

a3af3f0

add queryWithData to RemoteNode class

9d425da

factor node bootstrap out of repl.js

5d64716

add aleph query cli command

d598d2d

repl: don't swallow connection promise

ce5cc81

add --withObjects flag to inline data objects with query results

93e7c8f

allow connecting to peers by multiaddr string, use cached PeerInfo fr…

e828ccc

…om PeerBook

hack in support for /p2p/ multiaddrs during lookup

1c56efb

try to create config dir (default: ~/.mediachain/aleph) if it doesn't…

3274810

… exist

update shrinkwrap

5a39e7e

code style - rm semicolons

fe159fb

yusefnapora force-pushed the yn-query-with-data branch from cd8df36 to fe159fb Compare November 21, 2016 15:56

parkan reviewed Nov 21, 2016

View reviewed changes

update readme with aleph cli usage

5a701f5

[ci skip]

parkan approved these changes Nov 21, 2016

View reviewed changes

add comments for expandQueryResult helper, remove redundant base64 co…

d49eca4

…nversion

yusefnapora merged commit 12bcab6 into master Nov 21, 2016

yusefnapora deleted the yn-query-with-data branch November 21, 2016 19:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Queries with inline data (p2p only) #88

Queries with inline data (p2p only) #88

yusefnapora commented Nov 17, 2016

parkan commented Nov 17, 2016

yusefnapora commented Nov 17, 2016 •

edited

Loading

parkan Nov 21, 2016

parkan Nov 21, 2016

parkan Nov 21, 2016

yusefnapora Nov 21, 2016

parkan Nov 21, 2016

parkan Nov 21, 2016

parkan Nov 21, 2016

yusefnapora Nov 21, 2016

parkan Nov 21, 2016

Queries with inline data (p2p only) #88

Queries with inline data (p2p only) #88

Conversation

yusefnapora commented Nov 17, 2016

parkan commented Nov 17, 2016

yusefnapora commented Nov 17, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yusefnapora commented Nov 17, 2016 •

edited

Loading