Oracle: OracleService #1555

shargon · 2020-04-13T07:56:59Z

This service should receive all oracle transactions, request and response, also it should receive the p2p message.

Oracle service will process the downloads, the received signatures, and it will send to memPool the transactions only when the transactions are full signed.

TODO:

Send error if there are no agrement.

shargon · 2020-04-14T14:09:05Z

Please take a look to this PR in order to make the desired changes, we are very close to have a synchronized oracle :) @neo-project/core @erikzhang @neo-project/ngd-shanghai

src/neo/Oracle/OracleService.cs

roman-khimov · 2020-04-14T21:18:05Z

src/neo/Consensus/ConsensusContext.cs

@@ -283,7 +283,7 @@ public ConsensusPayload MakePrepareRequest()
            Span<byte> buffer = stackalloc byte[sizeof(ulong)];
            random.NextBytes(buffer);
            Block.ConsensusData.Nonce = BitConverter.ToUInt64(buffer);
-            EnsureMaxBlockSize(Blockchain.Singleton.MemPool.GetSortedVerifiedTransactions());
+            EnsureMaxBlockSize(Blockchain.Singleton.MemPool.GetSortedVerifiedTransactions(Snapshot));


Is there anything preventing response from being separated from the corresponding request here? I mean we may strip some transactions off of the block there and then we may get either request tx with no response or vice versa.

I think not yet

The request will wait for 2 conditions (any of them)

His response has already been iterated when iterating the mempool, (it means that it will enter in the same block).

There is already a result for that tx waiting in the states, (it means that the response has already entered a previous block)

You can see the code here
https://github.com/neo-project/neo/pull/1555/files#diff-eab33c87be022a91edcd35025f268e73R44-R81

I think IsReady ensures that we're getting both transactions as a result of GetSortedVerifiedTransactions(), that's fine, but what I'm talking about is EnsureMaxBlockSize filtering, it doesn't seem to be aware of transaction dependencies and we may lose one of them here.

roman-khimov · 2020-04-15T09:29:35Z

src/neo/Oracle/OracleService.cs

+                {
+                    if (request.IsCompleted)
+                    {
+                        // Done! Send to mem pool


I see multisignature transaction forming there and I'm wondering can we make it a generic mechanism not tied to OracleService? I mean having some protocol to glue together several signatures for regular multisignature transactions that OracleService could then reuse for its purposes.

At the moment we have to sign these transactions via some off-chain mechanism and then when all required signatures are in place push the result to the network, but with this mechanism we could push incomplete (in terms of required signatures) transactions to the network, let the network collect other incomplete transactions and then generate and push the result as soon as the required number of signatures is reached.

This is what already is being done here and if we're to extend it a little for regular transactions I think we can gain some interesting functionality. NeoFS actually could use it then for its purposes (we have a scenario where inner ring nodes need to agree on something and every node can sign for itself, but they need to create this multisignature transaction in the end).

It's a very good proposal, but maybe it's hard to solve, now we only considered to do this with oracle tx, but it's something very interesting, could you open a new issue for it?

belane · 2020-04-15T09:29:40Z

src/neo/Network/P2P/Payloads/OraclePayload.cs

-        private UInt256 _hash = null;
-        public UInt256 Hash
+        private UInt160 _hash = null;
+        public UInt160 Hash


Why 160? Just to reduce the size?

We use UInt256 only for IInventory

Maybe it should be a IInventory. What do you think @erikzhang ?

I think it should be IInventory, which can reduce repeated sending, just like normal tx/block/consensus message. We use inv message to push new message.

Receive oraclePayload

src/neo/Network/P2P/Payloads/Transaction.cs

Tommo-L · 2020-04-15T15:19:05Z

src/neo/Network/P2P/ProtocolHandler.cs

+
+            using var snapshot = Blockchain.Singleton.GetSnapshot();
+            if (payload.Verify(snapshot)) return;
+


Should we check whether the OraclePayload has a valid oracle request? Is it possible, one oracle node send a lot of useless OraclePayload to attack the network?

It's possible but it will be enqueued and discarded with time, but I will order it for count, and grouping by publicKey

It was improved the pool's sort in this order:

Sort by number of signatures

Sort by if has my signature or not

Sort by age

9e99eea

Also we can add a PoW in order to prevent the p2p spam, but I think that it's not required

Improve sorting response pool

shargon · 2020-04-16T12:21:21Z

I will merge it and it could be improved later, remember that you always can review the oracle branch.

We already tested our first oracle TX, in the same block :)

* Start pool * Prepare for neo-node * Fix double call * Prepare pool * Fix * Join oracle pool with oracle service * dotnet-format * Try to fix UT * UT pass * Fix UT * Fix p2p message * Unify collections * Relay p2p message * Clean code and fixes * Send tx to OracleService * RequestTx will wait for ResponseTx * Rename * Remove supervisor * Allow to put request, and response in the same block * Check the sender of OracleResponses * Remove OracleResponse message in OracleService * Organize code * Fix typo * Remove task count TODO * Remove Thread-safe TODO * Remove TODO * ResponseItem changes * Read the oracle contract on my response Receive oraclePayload * Clean code * Add Stop message * Save only one response for PublicKey/RequestTx Improve sorting response pool * Improve sort * Group sort methods in the same region * Ask for the request TX if I don't have it * Reorder code * Rename * First oracle TX

* first commit * format * Simplify code * add UT and add some feature * Fix bug * Add summary and fix bug * format * format * little change * little change * little change * change Fee & Fix bug * Optimize * add config class * FiX UT * Format * Fix UT * Fix format and Optimize * Add some UT * Fix bug and add UT * Update src/neo/SmartContract/Native/Oracle/OraclePolicyContract.cs Co-Authored-By: Luchuan <[email protected]> * little change * format * Update src/neo/SmartContract/Native/Oracle/OraclePolicyContract.cs Co-Authored-By: Shargon <[email protected]> * add check * change validator * Add double initialization check * add UT * Add UT * change UT * fill UT * up ut and fix NeoToken.GetValidators * remove console * reset NEO.getValidators * fix GetOracleValidators * Rename variable * Update src/neo/SmartContract/Native/Oracle/OracleContract.cs Co-Authored-By: Erik Zhang <[email protected]> * Update src/neo/SmartContract/Native/Oracle/OracleContract.cs Co-Authored-By: Erik Zhang <[email protected]> * Fix bug * change logic * Fix name * change to const * Add double initialization check and fix ut * optimize Co-authored-by: Shargon <[email protected]> Co-authored-by: Luchuan <[email protected]> Co-authored-by: Luchuan <[email protected]> Co-authored-by: Erik Zhang <[email protected]> Oracle Syscall (#1399) * Plugins from List to array * Move false to init * Fix UT * Refactor * UT * Change HTTP to HTTPS * UT Syscall * Remove Version * Remove HTTP from the syscall name * Check schema * Rename * Update src/neo/Oracle/OracleRequest.cs Co-Authored-By: Erik Zhang <[email protected]> * Fix abstract * Rename * Remove Body * Change order of members * Rename Neo.Oracle.Get Co-authored-by: Vitor Nazário Coelho <[email protected]> Co-authored-by: erikzhang <[email protected]> Co-authored-by: Belane <[email protected]> Oracle Syscall (#1399) * Plugins from List to array * Move false to init * Fix UT * Refactor * UT * Change HTTP to HTTPS * UT Syscall * Remove Version * Remove HTTP from the syscall name * Check schema * Rename * Update src/neo/Oracle/OracleRequest.cs Co-Authored-By: Erik Zhang <[email protected]> * Fix abstract * Rename * Remove Body * Change order of members * Rename Neo.Oracle.Get Co-authored-by: Vitor Nazário Coelho <[email protected]> Co-authored-by: erikzhang <[email protected]> Co-authored-by: Belane <[email protected]> Rename OracleResult->OracleResponse (#1516) Removed hash from the response Oracle Syscall (#1399) * Plugins from List to array * Move false to init * Fix UT * Refactor * UT * Change HTTP to HTTPS * UT Syscall * Remove Version * Remove HTTP from the syscall name * Check schema * Rename * Update src/neo/Oracle/OracleRequest.cs Co-Authored-By: Erik Zhang <[email protected]> * Fix abstract * Rename * Remove Body * Change order of members * Rename Neo.Oracle.Get Co-authored-by: Vitor Nazário Coelho <[email protected]> Co-authored-by: erikzhang <[email protected]> Co-authored-by: Belane <[email protected]> Rename OracleResult->OracleResponse (#1516) Removed hash from the response Oracle Service (#1517) * Oracle Service * Fix fee * Optimize load configuration * Fix akka message * Optimize * dotnet format * Fix sort * Oracle Service * Fix fee * Optimize load configuration * Fix akka message * Optimize * dotnet format * Fix sort * Apply rename * Small changes * Add UT * Fix Start Stop * dotnet-format * Migrate to HTTPS test * Change AKKA message * Tommo recomendations * Add comunication between OracleTx and UserTx * Advance of signature * Fix UT * dotnet-format * Advance of signature Fix UT dotnet-format * Fix UT * Remove duplica method * Reduce timeout time * Add log for fix UT * Add filters * Change port to 443 * Oracle Service * Fix fee * Optimize load configuration * Fix akka message * Optimize * dotnet format * Fix sort * Oracle Service * Fix fee * Optimize load configuration * Fix akka message * Optimize * dotnet format * Fix sort * Apply rename * Small changes * Add UT * Fix Start Stop * dotnet-format * Migrate to HTTPS test * Change AKKA message * Tommo recomendations * Add comunication between OracleTx and UserTx * Advance of signature * Fix UT * dotnet-format * Fix UT * Remove duplica method * Reduce timeout time * Add log for fix UT * Add filters * Change port to 443 * Merge changes * Change SSL Protocol * Merge remote-tracking branch 'neo-project/oracle-service' into oracle-service * Add UT certificate * Clean code * Add UT fix GetOracleMultiSigAddress (#1538) * fix GetOracleMultiSigAddress * First lowercase Co-authored-by: Shargon <[email protected]> Oracle: Tx comunication (#1540) * Tx comunication * Some fixes Transaction Version (#1539) Oracle: Allow wallets to add assert for the oracle response (#1541) * Add assert for each oracle request * Change to goto * Improve wallet asserts * Optimize * Fix URL type * dotnet format * Oracle: Tx comunication (#1540) * Tx comunication * Some fixes * Optimize OracleExecutionCache * Add assert for each oracle request * Change to goto * Improve wallet asserts * Optimize * Fix URL type * dotnet format * Optimize OracleExecutionCache * MakeTransaction UT Oracle: Set the right Tx version (#1542) * Set the right Tx version * Remove Error types from TX and VM * Neo.Oracle.Get now only return the result or null Oracle: P2P Response Signature Message (#1553) * Oracle Response Signature P2P * caches * rename private fields * remove comments * remove more comments and fix size * Clean code Co-authored-by: Shargon <[email protected]> Oracle: OracleService (#1555) * Start pool * Prepare for neo-node * Fix double call * Prepare pool * Fix * Join oracle pool with oracle service * dotnet-format * Try to fix UT * UT pass * Fix UT * Fix p2p message * Unify collections * Relay p2p message * Clean code and fixes * Send tx to OracleService * RequestTx will wait for ResponseTx * Rename * Remove supervisor * Allow to put request, and response in the same block * Check the sender of OracleResponses * Remove OracleResponse message in OracleService * Organize code * Fix typo * Remove task count TODO * Remove Thread-safe TODO * Remove TODO * ResponseItem changes * Read the oracle contract on my response Receive oraclePayload * Clean code * Add Stop message * Save only one response for PublicKey/RequestTx Improve sorting response pool * Improve sort * Group sort methods in the same region * Ask for the request TX if I don't have it * Reorder code * Rename * First oracle TX Oracle: Remove some TODOs (#1574) * Remove some todos * Remove one method * Reuse ManualWitness * dotnet format * Allow both tx in the same block without change order or fee * Fix p2p message serialization * Fix p2p * Fixes and IInventory * Fix akka message * Fix ut Fix conflicts Remove double call

shargon added 9 commits April 13, 2020 09:56

Start pool

5f6695a

Prepare for neo-node

cc015a5

Fix double call

4dfc867

Prepare pool

7bab169

Fix

1b70b56

Join oracle pool with oracle service

1e4a980

dotnet-format

0a9191e

Try to fix UT

9e40824

UT pass

ea5c2fc

shargon changed the title ~~Oracle: pool~~ Oracle: OracleService Apr 14, 2020

shargon marked this pull request as ready for review April 14, 2020 09:35

shargon mentioned this pull request Apr 14, 2020

[Oracles] Transactions #1274

Closed

shargon added 8 commits April 14, 2020 12:02

Fix UT

415eb6c

Fix p2p message

ce2307f

Unify collections

77f2d47

Relay p2p message

ff8a36c

Clean code and fixes

d50979b

Send tx to OracleService

ff0bd13

RequestTx will wait for ResponseTx

9e5d675

Rename

1a5d034

shargon added 5 commits April 14, 2020 16:20

Remove supervisor

9736e8c

Allow to put request, and response in the same block

4507f2e

Check the sender of OracleResponses

9f27f31

Remove OracleResponse message in OracleService

3edf3e1

Organize code

feb891b

roman-khimov reviewed Apr 15, 2020

View reviewed changes

belane reviewed Apr 15, 2020

View reviewed changes

shargon added 2 commits April 15, 2020 12:16

Fix typo

248b245

Remove task count TODO

1cfe115

shargon added 6 commits April 15, 2020 12:58

Remove Thread-safe TODO

f3d15f2

Remove TODO

7b72834

ResponseItem changes

d51d089

Read the oracle contract on my response

20bd59a

Receive oraclePayload

Clean code

4ddb77b

Add Stop message

ac37085

Tommo-L reviewed Apr 15, 2020

View reviewed changes

src/neo/Network/P2P/Payloads/Transaction.cs Show resolved Hide resolved

Tommo-L reviewed Apr 15, 2020

View reviewed changes

roman-khimov mentioned this pull request Apr 15, 2020

Network assistance for multisignature transaction forming #1573

Open

shargon added 7 commits April 15, 2020 20:44

Save only one response for PublicKey/RequestTx

9e99eea

Improve sorting response pool

Improve sort

c087f05

Group sort methods in the same region

2c68451

Ask for the request TX if I don't have it

1b9cb8a

Reorder code

c2561c3

Rename

4457d6a

First oracle TX

081e37b

shargon merged commit c32e6a4 into oracles Apr 16, 2020

shargon deleted the oracle-pool branch April 16, 2020 12:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Oracle: OracleService #1555

Oracle: OracleService #1555

shargon commented Apr 13, 2020 •

edited

Loading

shargon commented Apr 14, 2020

roman-khimov Apr 14, 2020

belane Apr 15, 2020

shargon Apr 15, 2020 •

edited

Loading

shargon Apr 15, 2020

roman-khimov Apr 15, 2020

roman-khimov Apr 15, 2020

shargon Apr 15, 2020

roman-khimov Apr 15, 2020

belane Apr 15, 2020

shargon Apr 15, 2020

shargon Apr 15, 2020

Tommo-L Apr 15, 2020

Tommo-L Apr 15, 2020

shargon Apr 15, 2020

shargon Apr 15, 2020 •

edited

Loading

shargon Apr 15, 2020

shargon commented Apr 16, 2020


		using var snapshot = Blockchain.Singleton.GetSnapshot();
		if (payload.Verify(snapshot)) return;

Oracle: OracleService #1555

Oracle: OracleService #1555

Conversation

shargon commented Apr 13, 2020 • edited Loading

shargon commented Apr 14, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shargon Apr 15, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shargon Apr 15, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shargon commented Apr 16, 2020

shargon commented Apr 13, 2020 •

edited

Loading

shargon Apr 15, 2020 •

edited

Loading

shargon Apr 15, 2020 •

edited

Loading