[Distributed] Ref counter thread safety

[krynju]

In various Dagger use cases, especially in an env with threads, issues have been observed where Future's were being set more than once.
Stacktraces, MWE and some investigation here JuliaParallel/Dagger.jl#267

So overall the suspicion was that Futures are being created with the same ids and are effectively being set twice when tasks are completed.
With this fix applied this is no longer happening as the refcounter is now threadsafe

However, we are still observing hangs and some other Dagger related issues in the same examples that were causing this future set twice issue, so we still need to investigate what is causing these.

Example stacktrace

Error in eager listener:
Future can be set only once
Stacktrace:
 [1] error(s::String)
   @ Base .\error.jl:33
 [2] put_future(rid::Distributed.RRID, v::MemPool.DRef, caller::Int64)
   @ Distributed C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.8\Distributed\src\remotecall.jl:590
 [3] call_on_owner(::Function, ::Distributed.Future, ::MemPool.DRef, ::Vararg{Any})
   @ Distributed C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.8\Distributed\src\remotecall.jl:514
 [4] put!(rr::Distributed.Future, v::MemPool.DRef)
   @ Distributed C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.8\Distributed\src\remotecall.jl:584
 [5] eager_thunk()
   @ Dagger.Sch c:\Users\krynjupc\.julia\dev\Dagger\src\sch\eager.jl:84
 [6] macro expansion
   @ c:\Users\krynjupc\.julia\dev\Dagger\src\processor.jl:154 [inlined]
 [7] (::Dagger.var"#53#54"{typeof(Dagger.Sch.eager_thunk), Tuple{}, NamedTuple{(:sch_uid, :sch_handle, :processor, :utilization), Tuple{UInt64, Dagger.Sch.SchedulerHandle, Dagger.ThreadProc, UInt64}}})()
   @ Dagger .\threadingconstructs.jl:178

@jpsamaroo

[vchuravy]

In isolation this changes look good, but as you mentioned might not suffice.

[Sacha0]

CI appears happy modulo FreeBSD; FreeBSD has been failing regularly lately so chances are it's negligible. (I wasn't sure what to make of the logs from a brief glance, but restarted for more data.)

[krynju]

Remaining bugs from the original issue (JuliaParallel/Dagger.jl#267) are addressed here JuliaParallel/Dagger.jl#281 and were happening due to Dagger code

[jpsamaroo]

This should be safe and trivial to backport; maybe it can make it into 1.7?

[KristofferC]

It's preferable with commit messages that are a bit more descriptive than "fix".

[krynju]

@KristofferC Should I force push a commit name change on this branch now that it's merged?

Also sorry for the inconvenience

[KristofferC]

Should I force push a commit name change on this branch now that it's merged?

No, just something to keep in mind for the future.

[vtjnash]

It is also more that vchuravy should have squashed this when merging and ensured the correctness of the message.

[vchuravy]

Mea culpa.