libp2p v0.18.0-rc1 and initial resource manager integration

[vyzo]

This upgrades libp2p to v0.18.0-rc1 (and pubsub to v0.6.1 fixing a leak while at it), and provides initial resource manager integration.

Note: there is a (non catastrophic) bug in our memory accounting somewhere in the stack, which results in log warnings about too much memory released.
This can be ignored for now, but I will try to get it fixed asap.

Edit: the bug has been fixed in go-yamux v3.0.2

---

The libp2p Resource Manager

With libp2p-v0.18.0-rc1 we introduce the resource manager, which accounts for and limits and resources used by the libp2p network stack.
See the resource manager repo for design notes and implementation.

This is the initial integration, whereby we initialize a resource manager and provide APIs for inspecting resource usage and setting limits.
In a subsequent pr we will also augment lotus network services to make their resources accountable by the resource manager by organizing them in logical services.

New APIs/Commands

We add the following APIs/cli commands:

• lotus net stat <scope> -- this allows you to inspect current resource usage for a scaope. The scope can be all, system, transient, svc:<service-name>, proto:<protocol-name>, or peer:<peer-name>
• lotus limit [--set] <scope> [<limit>] -- this allows you to inspect or set the current limit for a scope. The scope can be as above while the limit is a json representation of a limit; see below for the format.

Limit Configuration

The user can override the default limits by placing a limits.json file in .lotus.
This file has the following format, with all limits optional and retaining their default value if omitted:

{
  "System": <limit>    // system-wide limits; optional
  "Transient": <limit> // transient resource limits

 "ServiceDefault": <limit> // default service limits
 "ServicePeerDefault": <limit> // default per service peer limit
 "Service": { <service>: <limit> ...} // service specific limits
 "ServicePeer": { <service>: <limit> ...} // per service peer limits

 "ProtocolDefault": <limit> // default protocol limits
 "ProtocolPeerDefault": <limit> // default per protocol peer limit
 "Protocol": { <protocol>: <limit> ...} // protocol specific limits
 "ProtocoPeerl": { <protocol>: <limit> ...} // per protocol peer limits

 "PeerDefault": <limit> // default peer limits
 "Peer" : {<peer>: <limit> ...} // peer specific limits

 "Conn": <limit> // connection scope limits; you shouldn't have to modify this.
 "Stream": <limit> // stream scope limits; you shouldn't have to modify this.
}

Limit objects specify memory and stream/connection/fd limits. Memory can be either a fixed number (in bytes), a fraction (with min and max in bytes) of total of total memory, or a dynamically computed fraction (with min and max in bytes) of free memory at the time of check.
The object has the following format in json:

{
  // memory limit
  "Memory": <int> // fixed memory in bytes; if unset the memory is computed.
  "Dynamic": <bool> // whether the computed memory is static or dynamic
  "MemoryFraction": <float> // fraction of total/free to use 
  "MinMemory": <int> // min memory in bytes
  "MaxMemory": <int> // max memory in bytes

  // network limits
  "Streams": <int> // max total number of streams
  "StreamsInbound": <int> // max inbound streams
  "StreamsOutbound": <int> // max outbound streams
  "Conns": <int> // max total number of connections
  "ConnsInbound": <int> // max inbound connections
  "ConnsOutbound": <int> // max outbound connections
   "FD": <int> // max number of file descriptors
}

[vyzo]

The bug was in yamux memory accounting, fix is here: libp2p/go-yamux#77

[magik6k]

Looks mostly good. Not sure why linter complains. Also need more make gen

[vyzo]

That linter seems psychotic -- it is failing because of a test file in go-fil-markets that has broken because of mocknet api changes in go-libp2p.

[codecov]

Codecov Report

Merging #7969 (8a47a71) into master (04ce491) will decrease coverage by 0.14%.
The diff coverage is 6.15%.

@@            Coverage Diff             @@
##           master    #7969      +/-   ##
==========================================
- Coverage   39.31%   39.16%   -0.15%     
==========================================
  Files         658      660       +2     
  Lines       71134    71408     +274     
==========================================
+ Hits        27965    27966       +1     
- Misses      38348    38613     +265     
- Partials     4821     4829       +8     

Impacted Files	Coverage Δ
api/types.go	52.77% <ø> (ø)
cli/net.go	10.00% <0.00%> (-1.43%)	⬇️
markets/loggers/loggers.go	100.00% <ø> (+10.71%)	⬆️
node/builder.go	76.47% <ø> (ø)
node/impl/net/net.go	28.08% <ø> (ø)
node/impl/net/rcmgr.go	0.00% <0.00%> (ø)
node/modules/lp2p/libp2p.go	34.69% <ø> (ø)
node/modules/lp2p/rcmgr.go	47.05% <47.05%> (ø)
itests/kit/ensemble.go	91.24% <100.00%> (ø)
chain/stmgr/call.go	67.87% <0.00%> (-3.64%)	⬇️
... and 16 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 04ce491...8a47a71. Read the comment docs.