FWIW it does publish reasonably faster for me.

$> datalad -l debug publish --since= --to=datalad-public --missing=inherit -r
...
[DEBUG  ] Query repo: ['git', 'ls-files', '--stage', '-z', '-d', '-m', '--exclude-standard'] 
[DEBUG  ] Done query repo: ['git', 'ls-files', '--stage', '-z', '-d', '-m', '--exclude-standard'] 
[DEBUG  ] Done <AnnexRepo path=/mnt/datasets/datalad/crawl/openneuro (<class 'datalad.support.annexrepo.AnnexRepo'>)>.get_content_info(...) 
[DEBUG  ] Parsed version of PathRI '/mnt/datasets/datalad/crawl/openneuro/ds000001' differs from original PosixPath('/mnt/datasets/datalad/crawl/openneuro/ds000001') 
[DEBUG  ] Parsed version of PathRI '/mnt/datasets/datalad/crawl/openneuro/ds000002' differs from original PosixPath('/mnt/datasets/datalad/crawl/openneuro/ds000002') 
[DEBUG  ] Parsed version of PathRI '/mnt/datasets/datalad/crawl/openneuro/ds000003' differs from original PosixPath('/mnt/datasets/datalad/crawl/openneuro/ds000003') 
[DEBUG  ] Parsed version of PathRI '/mnt/datasets/datalad/crawl/openneuro/ds000005' differs from original PosixPath('/mnt/datasets/datalad/crawl/openneuro/ds000005') 
...

which takes notable amount of time to get through all hundreds. My wild guess is that it is probably due to diff coming up with full "records" for every subdataset, even when later they are not considered for actual action.

Codecov Report

Merging #4448 into maint will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##            maint    #4448      +/-   ##
==========================================
+ Coverage   89.62%   89.65%   +0.02%     
==========================================
  Files         288      288              
  Lines       40359    40361       +2     
==========================================
+ Hits        36173    36185      +12     
+ Misses       4186     4176      -10     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eac6cfc...2a3abc0. Read the comment docs.

This should avoid lengthy annotatepath in heavy hierarchies while trying to
look for possible changes in the wortree.

Yeah, I think comparing to HEAD rather than the working tree is preferable here, even without considering the improved performance. FWIW push compares to HEAD.

I think this would not be entirelly correct in case if subdataset is
progressed forward but change was not recorded in the superdaset yet, but it
might work as desired (publish modified in its forwarded on remote end as well)
which might be current behavior as well anyways (did not test).

I'm not sure I get what you're trying to say here, but I'd feel okay about assuming that no one is relying on the (mis)behavior of publish pushing out unregistered changes to subdatasets (if that is indeed what it does).

I think this would not be entirelly correct in case if subdataset is
progressed forward but change was not recorded in the superdaset yet, but it
might work as desired (publish modified in its forwarded on remote end as well)
which might be current behavior as well anyways (did not test).

I'm not sure I get what you're trying to say here, but I'd feel okay about assuming that no one is relying on the (mis)behavior of publish pushing out unregistered changes to subdatasets (if that is indeed what it does).

#!/bin/bash

set -eu
cd "$(mktemp -d ${TMPDIR:-/tmp}/dl-XXXXXXX)"

datalad create src
datalad create -d src src/subds

(
cd src
datalad create-sibling -r -s target ../target
# now modify subds without saving updated state in superdataset
touch subds/file
datalad save -d subds -m "added a file" -t 0.1
datalad publish --to target -r
)

git -C target status

On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   subds (new commits)

no changes added to commit (use "git add" and/or "git commit -a")

#!/bin/bash

set -eu
cd "$(mktemp -d ${TMPDIR:-/tmp}/dl-XXXXXXX)"

datalad create src
datalad create -d src src/subds

(
cd src
datalad create-sibling -r -s target localhost:$PWD/../target
# now modify subds without saving updated state in superdataset
touch subds/file
datalad save -d subds -m "added a file" -t 0.1
datalad publish --since= --to target -r
)

git -C target status

results in

[WARNING] fatal: bad revision 'target/master..HEAD' [diff(/home/yoh/.tmp/dl-AKim06i/src)] 
[WARNING] fatal: bad revision 'target/master..HEAD' [publish(/home/yoh/.tmp/dl-AKim06i/src)] 
publish(impossible): . (dataset) [fatal: bad revision 'target/master..HEAD']
[WARNING] fatal: bad revision 'target/master..HEAD' [diff(/home/yoh/.tmp/dl-AKim06i/src)] 
[WARNING] fatal: bad revision 'target/master..HEAD' [publish(/home/yoh/.tmp/dl-AKim06i/src)] 
publish(impossible): . (dataset) [fatal: bad revision 'target/master..HEAD']

and overall run failing... so I need to tune up this PR to work in those cases :-/ and probably ideally irrespectfully of --since been specified or not. heh heh

FTR: I have no objections to anything concerning publish(). In master it is a leaf command that isn't used outside the tests.

d'oh -- need to get back to this PR... current "performance" is simply not acceptable -- takes over 10 minutes to publish datasets.datalad.org whenever there is nothing to publish according to --since

those bad revision happen also with current maint version : so adding ..HEAD is not a problem -- it is just the fact that there is no remote branch yet (we have not pushed before). In such cases effective since should be None... will push a fix shortly