UTP causes memory leak #392
Comments
|
Do you have cgo enabled, and are using anacrolix/go-libutp? Can you provide any profiles? What kind of run time does it take for you to see the leak? |
|
cfg.DisableUTP=false |
|
I think cgo is enable |
|
Can you provide any profiles? What kind of run time does it take for you to see the leak? |
|
|
Here is some informations I saved before with debug/pprof |
|
@ucwong Thanks. I'm not sure what's going on there, you have a lot of goroutines blocked on unlocking the Client mutex? What versions of go-libutp and torrent are you on? Could you update to torrent v1.15.0? |
|
OK, I think i used the latest version, but i will test again |
|
Still OOM at last in utp mode |
|
Could it be that you're running lots of |
|
Do you have a read or write rate limiter set? |
|
... But it is ok for tcp protocol only, I use default set ,it maybe unlimited |
|
Any suggestions for utp settings different with tcp |
|
It's possible there are undocumented, upstream limits to how many connections you can have per utp socket. This information would be helpful:
|
|
@james-lawrence thank you for comment, I have test the new commit , the memory seems to be ok now, but the number of block and mutex is still increasing without OOM. I will keep running it for a bit longer time. |
|
The memory is increasing slower, but I think it still will be OOM after long time seeding with UTP |
|
I found 75 block here, ratelimitreader.go:43 |
|
Are you running with the block and mutex profilers enabled? I'm not sure the results mean what you infer them to mean. I don't have enough information to go on here. |
|
It shows where is blocked in the code, but I am not sure it is useful. It will oom when I seeding files for long time with UTP on. It is very good in only TCP mod. |
|
A heap profile after it has run for a long time before it crashes would be very helpful. |
|
@anacrolix I will make another test |
|
@ucwong thanks for sharing that. I would suggest not running with the block or mutex profiling enabled, those won't help with the memory leak as far as I'm aware. How many torrents do you think you have running at a given time? There's a backlog against the dial rate limiter, which I've not seen happen under normal circumstances. The heap profiles also aren't showing anything useful, I'm not sure why. Also what is your CPU usage like? I noticed a lot of CPU devoted to PEX, you might try disabling that. |
|
@ucwong do you have any more feedback? |
|
I am using it without UTP, It is ok till now. I am not sure why OOM by with UTP seeding |
|
It keeps printing this when I use utp |
|
@ucwong Does the local system support IPv6? |
package main
import (
"fmt"
"github.com/PuerkitoBio/goquery"
"github.com/anacrolix/torrent"
"log"
"net/http"
)
func main() {
config := torrent.NewDefaultClientConfig()
client, err := torrent.NewClient(config)
if err != nil {
log.Panic(client)
}
var magnets []string
for i := 1; i <= 13; i++ {
url := fmt.Sprintf("https://nyaa.si/?s=seeders&o=desc&p=%d", i)
response, err := http.Get(url)
if err != nil {
log.Fatal(err)
}
if response.StatusCode != 200 {
log.Fatalf("status code error: %d %s", response.StatusCode, response.Status)
}
doc, err := goquery.NewDocumentFromReader(response.Body)
if err != nil {
log.Fatal(err)
}
doc.Find("table > tbody > tr").Each(func(_ int, td *goquery.Selection) {
td.Find("td").Each(func(i int, a *goquery.Selection) {
if i == 2 {
magnet, _ := a.Find("a").Last().Attr("href")
magnets = append(magnets, magnet)
}
})
})
_ = response.Body.Close()
}
fmt.Printf("all length: %d \n", len(magnets))
for _, uri := range magnets {
_, err = client.AddMagnet(uri)
if err != nil {
log.Fatal(err)
}
}
select {}
}If UTP is enabled, memory is quickly depleted, If disabled, it takes about 10 minutes to run out. (~1000 torrent) ubuntu 20.04 / 32G Ram / go 1.16 / lib 1.15.1 / ipv6 and ipv4 |
|
Could subscribers to this issue try again with a recent version of anacrolix/torrent? |
Any changes about this ? |
|
Nothing specific but there has been a lot of churn throughout anacrolix/torrent. |
OK, I am running a long time test and hope it will be fine |
|
It seems better than before but memory can not be recycled till OOM in utp only mode |
|
@ucwong do you think this is fixed? |
|
I think I'm observing this issue when I have seeding enabled. Go reports ~85MB of memory in use, but the process is using ~1.25GB. |
|
A large memory leak was fixed in anacrolix/go-libutp@0430340. This is included in anacrolix/torrent v1.39.1 and later. Further fixes may occur from anacrolix/go-libutp#20, but I expect that the above fix handled most issues. It also includes a lot of performance improvements around uTP handling. |
util got OOM when i use utp enable
The text was updated successfully, but these errors were encountered: