No channel candidates - no channels

[ghost]

I'm running three nodes with CLN v24.02.1 and CLBOSS v0.13.

While the first node started opening channel right away, the other two nodes don't do anything.

Here's the output of one of them:

´´´
2024-03-22T04:55:38.313Z INFO plugin-clboss: ChannelCreationDecider: High fees, but 54.0433% of our funds are onchain, which is above our limit of 25%.Will create channels worth 6026983000msat (3000000
0msat reserved for onchain actions).
2024-03-22T04:55:41.535Z INFO plugin-clboss: ChannelCreator: Insufficient channel candidates, will solicit more.
´´´

ChannelFinderByDistance/Popularity are both running:

2024-03-22T04:55:46.233Z INFO    plugin-clboss: ChannelFinderByDistance: Preinvestigating: 035bf7d568f3bb3956c6ea8bd1c455c229c23ff85037fdd8a17baa9682045b5aa7(46234msat), 02a71032ddb75da53d0b447225c3bed8ee
2e2b39da622e11a89d58fe5b5b003c8c(31394msat), 0350587f325dcd6eb50b1c86874961c134be3ab2b9297d88e61443bb0531d7798e(30394msat), 0278e2e2f0fbfa727863e4d8f4303f27bce8eca7f4e55b7d7239575bb15b2a5437(30390msat), 0
339d25472ae71dcf7060fdf5a0a366ac97205ef3336cfa4ddee4e18834d1bbfa3(11326msat), 03f983dcabed6baa1eab5b56c8b2e8fdc846ab3fd931155377897335e85a9fa57c(10968msat), 02ba62f2cf65a10b9867477f266e67ef7e5f34b8b8b1289
16497bf54e0603a4693(3378msat), 0247024d82d2c9a87e330cbd9dfd2ff8ead0fcc7e6fe5646fceb587af30ba3325c(3378msat),  ... 32 more
2024-03-22T04:55:46.308Z INFO    plugin-clboss: Connector: failed to connect to 035bf7d568f3bb3956c6ea8bd1c455c229c23ff85037fdd8a17baa9682045b5aa7
2024-03-22T04:55:46.349Z INFO    plugin-clboss: Connector: connected to 02a71032ddb75da53d0b447225c3bed8ee2e2b39da622e11a89d58fe5b5b003c8c
2024-03-22T04:55:46.349Z INFO    plugin-clboss: ChannelCandidatesPreinvestigator: Proposing 02a71032ddb75da53d0b447225c3bed8ee2e2b39da622e11a89d58fe5b5b003c8c (patron 02f5c3900821c8245d7a8991a39188060da0f
65e6133a9ceb0415e09ed572255e1)
2024-03-22T04:57:51.448Z INFO    plugin-clboss: ChannelFinderByPopularity: Random selection (by popularity): 021d2436cab847373a4212bf6d754ead5304f5d0791479643893a837b295f3441c (69 peers)
2024-03-22T04:57:58.636Z INFO    plugin-clboss: Connector: failed to connect to 02c21c9d5a0a477e5038603eba4cec25dc54a8cb113b8022277f20ef82b6d01fc6
2024-03-22T04:58:01.512Z INFO    plugin-clboss: Connector: failed to connect to 0350587f325dcd6eb50b1c86874961c134be3ab2b9297d88e61443bb0531d7798e
2024-03-22T04:58:23.817Z INFO    plugin-clboss: Connector: failed to connect to 0278e2e2f0fbfa727863e4d8f4303f27bce8eca7f4e55b7d7239575bb15b2a5437
2024-03-22T04:58:23.817Z INFO    plugin-clboss: Connector: failed to connect to 0224ee56bfe91ebb9038fd8120fc2343fca7cbf0aea3aa43caca384dfaa56901af
2024-03-22T04:58:27.509Z INFO    plugin-clboss: Connector: connected to 02d904870a5655101789211614af38508fe03e4d746886264a84df4f6b12155b2b
2024-03-22T04:58:27.509Z INFO    plugin-clboss: ChannelCandidatesPreinvestigator: Proposing 02d904870a5655101789211614af38508fe03e4d746886264a84df4f6b12155b2b (patron 021d2436cab847373a4212bf6d754ead5304f
5d0791479643893a837b295f3441c)
2024-03-22T04:58:30.021Z INFO    plugin-clboss: Connector: failed to connect to 0339d25472ae71dcf7060fdf5a0a366ac97205ef3336cfa4ddee4e18834d1bbfa3
2024-03-22T04:58:30.609Z INFO    plugin-clboss: Connector: connected to 03f983dcabed6baa1eab5b56c8b2e8fdc846ab3fd931155377897335e85a9fa57c
2024-03-22T04:58:30.609Z INFO    plugin-clboss: ChannelCandidatesPreinvestigator: Proposing 03f983dcabed6baa1eab5b56c8b2e8fdc846ab3fd931155377897335e85a9fa57c (patron 02f460ae6d3d3e104f8afe520ae0cff3d94c3
5c2ba8df66da89f3c8006a265b90a)
2024-03-22T04:58:30.743Z INFO    plugin-clboss: Connector: connected to 02ba62f2cf65a10b9867477f266e67ef7e5f34b8b8b128916497bf54e0603a4693
2024-03-22T04:58:30.744Z INFO    plugin-clboss: ChannelCandidatesPreinvestigator: Proposing 02ba62f2cf65a10b9867477f266e67ef7e5f34b8b8b128916497bf54e0603a4693 (patron 03a09f56bba3d2c200cc55eda2f1f069564a9
7c1fb74345e1560e2868a8ab3d7d0)
2024-03-22T04:59:49.118Z INFO    plugin-clboss: AutoDisconnector: 02087d9151cb3afa60d210ccc7379fb4376d1c67c810290e32acbfecdfe5716e5d, 02ad7c3fdd57dd30a6894178f0a9836055cb1ac1957c33117d0129144aeb56e2e0, 03
f983dcabed6baa1eab5b56c8b2e8fdc846ab3fd931155377897335e85a9fa57c, 02ba62f2cf65a10b9867477f266e67ef7e5f34b8b8b128916497bf54e0603a4693
2024-03-22T05:00:19.207Z INFO    plugin-clboss: InternetConnectionMonitor: online.
2024-03-22T05:10:19.192Z INFO    plugin-clboss: InternetConnectionMonitor: online.
2024-03-22T05:18:28.297Z INFO    plugin-clboss: ChannelCandidateInvestigator: No candidates.

So CLBOSS seem to be proposing channel, however lightning-cli clboss-status delivers no candidates:

{
   "channel_candidates": [],
   "closed_peer_complaints": {},
   "incoming_capacity_swapper": {
      "status": "idle",
      "message": "Newly-started"
   },
   "internet": {
      "connection": "online",
      "checking_connectivity": false
   },
   "lnfee": {
      "026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2": {
         "median_base": 0,
         "median_proportional": 77,
[...]

I wonder what I'm missing here. Am I being to impatient?

Do you have any channel_candidates in your running instance?

[ghost]

I forgot: I manually opened private channels between these three nodes and public channels to other bigger nodes.

[ksedgwic]

I'm seeing this behavior as well. Starting to fill in my mental channel candidate flow diagram (it's pretty complicated). Here some random hypothesis:

1. the probing is failing - maybe it doesn't think the nodes are "good"
2. negotiations fail (maybe due to changes in feerate assumptions etc) and then the failure leads to avoiding the nodes in the future
3. I've been wondering various config changes I've been playing with could be affecting suitability (ie, max-locktime-blocks or max-concurrent-htlcs)

The fact that others are seeing this recently is a big clue.

I'm investigating but don't have a prognosis yet. Please post theories or discoveries here!

[ksedgwic]

A clue! While working on something else, noticed this go by:

Tue 2024-04-02 11:11:03 PDT home4 lightningd[733270]: 2024-04-02T18:11:03.312Z INFO    plugin-clboss: ChannelCandidatesPreinvestigator: Proposing 028ec70462207b57e3d4d9332d9e0aee676c92d89b7c9fb0850fc2a24814d4d83c (patron 031c64a68e6d1b9e50711336d92b434c584ce668b2fae59ee688bd73713fee1569)
Tue 2024-04-02 11:11:03 PDT home4 lightningd[733270]: 2024-04-02T18:11:03.313Z DEBUG   lightningd: Calling rpc_command hook of plugin clboss
Tue 2024-04-02 11:11:03 PDT home4 lightningd[733270]: 2024-04-02T18:11:03.313Z DEBUG   plugin-clboss: Rpc out: getroute {\"id\": \"031c64a68e6d1b9e50711336d92b434c584ce668b2fae59ee688bd73713fee1569\", \"fromid\": \"028ec70462207b57e3d4d9332d9e0aee676c92d89b7c9fb0850fc2a24814d4d83c\", \"msatoshi\": \"1msat\", \"riskfactor\": 10, \"exclude\": [\"03942f5fe67645fdce4584e7f159c1f0a396b05fbc15f0fb7d6e83c553037b1c73\"], \"fuzzpercent\": 0, \"maxhops\": 3}
Tue 2024-04-02 11:11:03 PDT home4 lightningd[733270]: 2024-04-02T18:11:03.317Z DEBUG   lightningd: Plugin clboss returned from rpc_command hook call
Tue 2024-04-02 11:11:03 PDT home4 lightningd[733270]: 2024-04-02T18:11:03.321Z DEBUG   plugin-clboss: Rpc in: getroute {\"id\": \"031c64a68e6d1b9e50711336d92b434c584ce668b2fae59ee688bd73713fee1569\", \"fromid\": \"028ec70462207b57e3d4d9332d9e0aee676c92d89b7c9fb0850fc2a24814d4d83c\", \"msatoshi\": \"1msat\", \"riskfactor\": 10, \"exclude\": [\"03942f5fe67645fdce4584e7f159c1f0a396b05fbc15f0fb7d6e83c553037b1c73\"], \"fuzzpercent\": 0, \"maxhops\": 3} => error {\"code\":-32602,\"message\":\"unknown parameter: msatoshi, this may be caused by a failure to autodetect key=value-style parameters. Please try using the -k flag an...

[ksedgwic]

More info, the first incident of the unknown parameter: msatoshi error was on 24-02-28T02:23:29.05 which is right after updating to v23.11-440 (which really was v24.02, there was a problem w/ the tag ...)

The prior version (which did not have the problem) was v23.11

[ksedgwic]

This looks to be working well on my systems, finding candidates, opening channels, ...