bootstrapping leads to no connection due to many dead peers

[upperwal]

During testing the bootstrap node is populated with many dead peers. As a result we don't get a healthy peer to connect too.

23:09:22.224 DEBUG svc-bootst: Found peer: {QmPmAGEYy3WbC6q7MzGkoXt9qXdv7PjjChrFCgyVY8VyPF: []} connection.go:81
23:09:22.224 DEBUG svc-bootst: Found peer: {QmNv5MX4o4YAqsUSZiqx9rVcaroVa6KTcAjkuDGYwTsYPK: []} connection.go:81
23:09:22.224 DEBUG svc-bootst: Found peer: {QmeLepdbZoAVL4SbMJhgSLwutH9H1BV96zmGGwbvzmTPSb: []} connection.go:81
23:09:22.224 DEBUG svc-bootst: Found peer: {QmbSXsYy2JpxgZZzqH6MSkAH1jxvSrAyoZNJD7Hn7f9f3P: []} connection.go:81
23:09:22.224 DEBUG svc-bootst: Found peer: {QmcTNbyvYWa6Ui8EXku2TgXBgqauK1dx3wVNUGeZ2PdisM: []} connection.go:81
23:09:22.224 DEBUG svc-bootst: Found peer: {QmZaXMgBa494QZY5XuMSL7mnKYdRyuZzaJSQsuU3bdqF85: []} connection.go:81
23:09:22.225 WARNI svc-bootst: Connection failed: failed to dial QmbSXsYy2JpxgZZzqH6MSkAH1jxvSrAyoZNJD7Hn7f9f3P: no addresses connection.go:87
23:09:22.225 WARNI svc-bootst: Connection failed: failed to dial QmNv5MX4o4YAqsUSZiqx9rVcaroVa6KTcAjkuDGYwTsYPK: no addresses connection.go:87
23:09:22.224 WARNI svc-bootst: Connection failed: failed to dial QmeLepdbZoAVL4SbMJhgSLwutH9H1BV96zmGGwbvzmTPSb: no addresses connection.go:87
23:09:22.224 WARNI svc-bootst: Connection failed: failed to dial QmPmAGEYy3WbC6q7MzGkoXt9qXdv7PjjChrFCgyVY8VyPF: no addresses connection.go:87
23:09:22.225 WARNI svc-bootst: Connection failed: failed to dial QmcTNbyvYWa6Ui8EXku2TgXBgqauK1dx3wVNUGeZ2PdisM: no addresses connection.go:87
23:09:22.225 DEBUG svc-bootst: Found peer: {QmVzDFh9iRh5g1DvYW5FCrd1DMETveUuR4HswCxEsAqGhu: []} connection.go:81
23:09:22.225 DEBUG svc-bootst: Found peer: {QmbWMvkxknPhYjNAJB4jtftx36Co5RqHUFfBi9Uk89N7aL: []} connection.go:81
23:09:22.225 WARNI svc-bootst: Connection failed: failed to dial QmZaXMgBa494QZY5XuMSL7mnKYdRyuZzaJSQsuU3bdqF85: no addresses connection.go:87
23:09:22.225 DEBUG svc-bootst: Found peer: {QmU9TwVrjqwvFdXsAa2QGxRvGVWZwXnuJpng7nVf6hiFmW: []} connection.go:81
23:09:22.225 DEBUG svc-bootst: Found peer: {Qmajpxt5vNAVAUTcmLG26zrYMXD45URDo7EMZY8MNVkJDS: []} connection.go:81
23:09:22.225 WARNI svc-bootst: Connection failed: failed to dial QmVzDFh9iRh5g1DvYW5FCrd1DMETveUuR4HswCxEsAqGhu: no addresses connection.go:87
23:09:22.225 WARNI svc-bootst: Connection failed: failed to dial QmbWMvkxknPhYjNAJB4jtftx36Co5RqHUFfBi9Uk89N7aL: no addresses connection.go:87
23:09:22.225 WARNI svc-bootst: Connection failed: failed to dial QmU9TwVrjqwvFdXsAa2QGxRvGVWZwXnuJpng7nVf6hiFmW: no addresses connection.go:87
23:09:22.225 DEBUG svc-bootst: Found peer: {QmXfDRfnWjqGYD5ZQ8iiAjsDYBibtBsPCA1CJew9KMg8kc: []} connection.go:81
23:09:22.225 DEBUG svc-bootst: Found peer: {QmTUcnRvfaf15G5uouskCrqiGGtRQKkn4eZ8NwQwHuU2Ni: []} connection.go:81
23:09:22.226 DEBUG svc-bootst: Found peer: {QmabH7BsDANPX3ghHwCnKuxFXLJF3fv7bBuR2ZTb4z2HPw: []} connection.go:81
23:09:22.226 WARNI svc-bootst: Connection failed: failed to dial Qmajpxt5vNAVAUTcmLG26zrYMXD45URDo7EMZY8MNVkJDS: no addresses connection.go:87
23:09:22.226 DEBUG svc-bootst: Found peer: {QmP7GrwHUNbBrW8FjMvZ9MusCUWScz1VtKqd4ohbCu1rim: []} connection.go:81
23:09:22.226 WARNI svc-bootst: Connection failed: failed to dial QmXfDRfnWjqGYD5ZQ8iiAjsDYBibtBsPCA1CJew9KMg8kc: no addresses connection.go:87
23:09:22.226 DEBUG svc-bootst: Found peer: {Qmbu4H9uVacrz2L5PRBEQkeQsS7TrZZHZBRPGtNMGKT1Ti: []} connection.go:81
23:09:22.226 WARNI svc-bootst: Connection failed: failed to dial QmTUcnRvfaf15G5uouskCrqiGGtRQKkn4eZ8NwQwHuU2Ni: no addresses connection.go:87
23:09:22.226 WARNI svc-bootst: Connection failed: failed to dial QmabH7BsDANPX3ghHwCnKuxFXLJF3fv7bBuR2ZTb4z2HPw: no addresses connection.go:87
23:09:22.226 WARNI svc-bootst: Connection failed: failed to dial QmP7GrwHUNbBrW8FjMvZ9MusCUWScz1VtKqd4ohbCu1rim: no addresses connection.go:87
23:09:22.226 WARNI svc-bootst: Connection failed: failed to dial Qmbu4H9uVacrz2L5PRBEQkeQsS7TrZZHZBRPGtNMGKT1Ti: no addresses connection.go:87

Solution
Auto Bootstrap.

[upperwal]

Calling FindPeers any number of times yield same set of peer ID's. So, auto bootstrapping isn't working.

A better solution is to create a new bootstrap protocol (stream) with its own .proto file. This bootstrap protocol can support queries like

1. Get me peers from some geo location bound.
2. Get peers from a collection of topics (topic => []peers).
3. Get me peers from some topic which are healthy and reliable. Health and reliability can be defined in terms of peers processed, up time, etc
4. ...