Big repo causes resource hog hangs

[Kubuxu]

I can't ls a hash from files API, nor name publish it, although I can get object for it.

See: https://ipfs.io/ipns/bin.ipfs.ovh/#QmbXUsz2aYebJvF2GE5i8j3WEP9NhWpQ1p6F6QWRosEyE8.txt

[Kubuxu]

Seems connected with #2407.
ipfs cat [some hash] also hangs using massive amount of disk bandwidth.

[whyrusleeping]

could you post a stack dump from the daemon while this hang is happening?

[Kubuxu]

Here is stack trace: https://gist.github.com/Kubuxu/7746e2770650796ab043

Also this is quite interesting:

1d [kubuxu@vs1:~] $ ipfs files ls /
ovh-kubuxu
public
1d [kubuxu@vs1:~] $ ipfs files ls /public
^C
Error: request canceled
1d [kubuxu@vs1:~] 1 $ ipfs files ls /
^C
Error: request canceled
1d [kubuxu@vs1:~] 1 $ 

[whyrusleeping]

In the debug/vars log you sent me, there were a lot of calls to HasBlock, only 5 actual Get calls, and one call to Get that errored. I just got my raid box back and set up so I can start trying out larger tests like this. I would run this with the log levels raised to info ( ipfs log level all info ) and see if anything interesting shows up there.

[whyrusleeping]

@Kubuxu is this improved by the bloom filter code?

[Kubuxu]

I traced it to the GC code. Let me find it.

Bloom won't help as what happens we recalculate size of the repo on every call or something ridiculous like that.

[Kubuxu]

Here: #2174

It is slow in case of 250MiB, looks like hang in case of multiple GiB.

At least I think it might be the same issue, I might be wrong.

EDIT: It might be improved by bloom but it shouldn't be a problem in the first case, why are we running so many Has calls when we try to access something.

[whyrusleeping]

This should be partially resolved, ipfs cat no longer tries to run a gc