Investigate whether Tasks plugin affects editing speed in large vaults - Answer: it does, and will be worked on soon.

[claremacrae]

In conversation with @schemar I learned that the way that Tasks tracks edits to Obsidian notes is to reload all files. And that has been fast enough.

There have been some anecdotal comments that maybe Obsidian with the Tasks plugin uses more CPU - and that editing can be laggy.

This issue is to investigate if performance or CPU is a problem, and if it is, then to resolve the issue.

Possible Solution

Refactor the code so that when a note is edited, only that note is re-read, and not the whole vault.

This will need to be sure detect edits made in external editors.

[claremacrae]

Example observation from @elight in Discord

Is it just me or does Tasks eat up a fair amount of CPU? I'm noticing more lag in my data entry in my vault, since enabling it.
I'll check more tomorrow AM when I'm on my Mac as opposed to my iPad.

[claremacrae]

From me:

I think it may not be just you,
I think it may be worse on iPad and iPhone than on Mac
My efforts to get a solid repro have not succeeded, hence wanting to start logging their setups, what plugins they have enabled etc.

TODO Add a note about the size of my vaults - numbers of files and tasks

I must admit that I have not yet thoroughly tested it with only the Tasks plugin enabled, and I'm so dependent on CustomJS as well.

[claremacrae]

@schemar I hope I've represented correctly what you said. If you have a mo, any info about the relevant bits of code would really be appreciated!

[schemar]

Thank you for tracking this, @claremacrae! The details a bit different, though. When a note changes, the cache only refreshes the tasks from that note, not all notes.

Here, the cache subscribes to the vault changes on a per file level and calls indexFile for changes: https://github.com/schemar/obsidian-tasks/blob/09758e51411ca124c4725b00ada023220314eccd/src/Cache.ts#L115

Here, the cache clears all tasks from that file before re-processing the file: https://github.com/schemar/obsidian-tasks/blob/09758e51411ca124c4725b00ada023220314eccd/src/Cache.ts#L221

However, what is currently non-optimized are propagation of changes to the cache's subscribers. For example, each query view subscribes to the cache to be informed of changes to update the view when tasks change. The Cache::notifySubscribers() method always sends all tasks to all subscribers: https://github.com/schemar/obsidian-tasks/blob/09758e51411ca124c4725b00ada023220314eccd/src/Cache.ts#L93

Potential optimization:
Only send the tasks of the changed note. For that, all subscribers would need a map of note->tasks to update only partially.

[claremacrae]

Oooh, the internal label is very nice. Thank you.

And thank you for all the info, @schemar!

[elight]

Confirmed this morning that I don't notice any lag on data entry/manipulation on my i7 Mac mini but significant data entry/manipulation lag on my M1 iPad Pro. I'll experiment further on my iPad to attempt to determine if it is in fact the Task plugin or something else. Early signs point toward my recent introduction of the Task plugin to my Vault.

[elight]

And thanks again for reporting, @claremacrae !

[YannickTemplay]

Hello,
Tasks (1.6.0) make my quite large vault (21k notes) almost unusable with lags of 5 to 10 seconds in the middle of edits. If I disable tasks then the performance switch back to almost instantaneous. I am on an Apple M1 with 16MB of Ram...

I am not sure how many tasks I have but if counting "dones" then quite a lot.

Maybe you could add an option to the tasks queries are only updated once when a page loads or something ?

The tasks plugin is central to my Obsidian usage but right now I am down to disabling it and reenabling it every time I need to see my tasks :-(

Thanks for any help !

[claremacrae]

@YannickTemplay Sorry to hear that. What version of Tasks were you on previously, and what other plugins do you have enabled?

[claremacrae]

@YannickTemplay any extra info that you can supply would be really helpful. When you say M1 I presume you mean a laptop rather than iPad?

For example:

is it all edits, or edits in files with tasks?

Does it differ if you are Live Preview or Source mode?

Is it worse if your edits are further from the start of the file?

[YannickTemplay]

Yes, Mac M1

All notes (but I think its much worse on notes with tasks queries)

See-> https://www.loom.com/share/c92f3a4b6339451d9ca62b0241d21ff9

[aubreyz]

Yes, Mac M1

All notes (but I think its much worse on notes with tasks queries)

See-> https://www.loom.com/share/c92f3a4b6339451d9ca62b0241d21ff9

Are any of the queries on the page dataview queries on tasks?

I had the same problem with task queries but more prominently with dataview ones. One thing I did not understand was why it had nothing to do with the mode (edit, live preview, reading). So two observations:

a) Why would searches ever refresh (for either plugin) when in edit mode -- but I can confirm that this definitely seemed to be happening). Only one pane open.

b) It seemed to happen when the output of the query was large (i.e. queries that produced many hundreds of hits that had to be rendered), not anything do do with the complexity of the query or the quantity of underlying data (bizarrely even if the result was not actually displayed in HTML). Whatever the case, at least in my hands, the intermittent severe lags on editing are a function of the page contents and not the vault itself.

I cannot see why tasks (or dataview for that matter) should be trying to process anything at all ever except when exiting a page (either by going to another page, or going from edit mode to preview). Arguably it should even do no-processing in live preview. It should certainly not do anything at all at each and every autosave of a file while editing things.

I'm on Windows 10

[YannickTemplay]

The video I sent you has no queries at all. Just a plain new note. And no other notes were opened simultaneously.

Of course, the lags are much worse on pages with a query or multiple queries, but the lag is still there on pages without queries so I think there is indeed a problem with some "refresh" that takes place in the background unnecessarily.

Typical queries I use on my daily note page look like that...

tasks
not done
due on {{date:YYYY-MM-DD}}
group by filename

I often also use in projet notes queries like this...

dataviewjs
const name = dv.current().file.name;
dv.taskList(dv.pages().file.tasks.where(t => !t.completed && t.text.includes(name)));

[YannickTemplay]

Could the problem come from the dataview Javascript queries?

I copied this query from Excalidraw Zsolt.

dataviewjs
const name = dv.current().file.name;
dv.taskList(dv.pages().file.tasks.where(t => !t.completed && t.text.includes(name)));

In his example, I think he mentioned that he put a comment with a "random number" in every query. I did not understood why so I did not do that. But maybe it's related to dataview refresh or something?

But again those dataview JS queries are not on those empty new pages so I don't understand why it would slow things down and when I turn off TASKS is works again while I keep DATAVIEW active... so it's not that I think.

[aubreyz]

To add to my last message -- the page that caused the most serious problems in my hands -- with editing lags in pure edit mode: Was a page of examples I use which has a host of (dozens of) personal dataview and tasks examples I have collected. This file is almost un-editable (but can be easily previewed/rendered).

I know this sounds bizarre, but there is something fishy going on under the hood.

[claremacrae]

Ah right, knowing that you also have dataview enabled, it would be good to understand if there is any interaction between your dataview and tasks setups.

Is there any chance you could disable dataview and restart Obsidian, and explore the behaviour, please?

[YannickTemplay]

Yes, the lags seem to come when both TASKS and DATAVIEW are enabled together...

Just TASKS = OK

Just DATAVIEW = OK

TASKS + DATAVIEW = lags

[claremacrae]

This is all tremendously helpful!

As much as possible, it would be really good to get approximate measures of timings with just dataview and tasks enabled an nothing else.

(Someone mentioned Excalidraw too. The number of potential interactions between plugins goes up exponentially with the number of plugins enabled.)

[aubreyz]

OK I should qualify my previous comments. The previous file I had containing a large amount of example dataview and task searches, and which (6 weeks ago) produced massive lags in pure edit mode, no longer does that. So perhaps some update to core obsidian has fixed it for me. I also use excalidraw occasionally (not in these pages though).

The page does still have massive problems on editing in live preview mode (much less of a problem, and perhaps understandable, but probably unnecessary). It is not clear why every search on a page should be re-done every few characters of typing if text that is being typed is completely outside of any search.

[claremacrae]

Thanks @aubreyz

So perhaps some update to core obsidian has fixed it for me.

Perhaps., but I'm not sure ... Every time I thought I had a good reproduction of the problem, and restarted Obsidian to verify it, the problem would not occur. And then some time later (hours or so) it would suddenly be slow again.

As a developer, things failing intermittently makes it almost impossible to work reliably on the bug, because you never know whether any individual change you made fixed the problem, if you have to keep testing for multiple hours to even be sure you
really fixed it.

I also use excalidraw occasionally (not in these pages though).

Do you mean you turn excalidraw off when you are not using it?

The thing is that some plugins may do work behind the scenes, even if they are not used on the particular page you are editing.... Hence my request for testing to be done with other plugins turned off...

The page does still have massive problems on editing in live preview mode (much less of a problem, and perhaps understandable, but probably unnecessary). It is not clear why every search on a page should be re-done every few characters of typing if text that is being typed is completely outside of any search.

See the comment #697 (comment) above, from the author of this plugin, for comments about that.

Once we get to a reliable reproduction of this issue on a development machine, we can look at optimising that code.

[aubreyz]

I also use excalidraw occasionally (not in these pages though).

Do you mean you turn excalidraw off when you are not using it?

No I don't switch it off. But no point trying that now, unless I can reproduce the issue again on editing (not on rendering)

[aubreyz]

I think a related issue is the autosave interval in the obsidian core. This is (as far as I know) fixed at about two seconds, and is not changeable.

So if something is triggered on a save during editing, this is massively magnified. It was fairly obvious before when I was having files causing these problems, that editing lag was cycling at about that frequency -- so if I just typed xxxxxxxxxxxxxxxxxxxxxx I got about two seconds of typing in, before having to wait about 10 seconds.

It also creates problems for file versioning (by say Dropbox) when there are just thousands of versions.

It would be quite nice if the Obsidian developers allowed this to be altered within some sensible limits (say up to 30 seconds or when the pane was exited or previewed). This would also help to debug these types of issues because the timing aspect would become more clear.

[elight]

My steps to replicate:

1. Open Obsidian on my M1 iPad Pro.
2. Ensure DataView and Tasks are enabled
3. Open my Daily Note (Contains several Tasks queries and a DataView query)
4. Begin typing into the note,
5. Switch off DV
6. Repeat Steps 1-4 
7. Switch on DV
8. Switch off Tasks
9. Repeat Steps 1-4
10. Switch on Tasks
11. Open a new note with no queries
12. Type into the new note.

Results

Steps	Observations
Steps 1-4	Input lag was noticed. After typing a few characters, input seemed to buffer, rendering approximately every 2 seconds.
Steps 5-6	Same lag was observed.
Steps 7-9	No lag was observed
Steps 10-12	No lag was observed.

This lag has a significant negative impact to usability on iPadOS. I haven't interacted with the iOS version enough lately to comment.

[elight]

I think a related issue is the autosave interval in the obsidian core. This is (as far as I know) fixed at about two seconds, and is not changeable.
...

The observed 2 seconds aligns with my observations on iPadOS.

[YannickTemplay]

I wonder if all of you who have the lag issue use dataview JS queries too ?

Could it be related ?

dataviewjs
const name = dv.current().file.name;
dv.taskList(dv.pages().file.tasks.where(t => !t.completed && t.text.includes(name)));

[elight]

I do use dataviewJS but I don't on that particular note, just DV's query language.

On May 31, 2022, Vipercat ***@***.***> wrote: I wonder if all of you who have the lag issue use dataview JS queries too ? Could it be related ? dataviewjs const name = dv.current().file.name; dv.taskList(dv.pages().file.tasks.where(t => !t.completed && t.text.includes(name)));

— Reply to this email directly, view it on GitHub <#697 (comment) 1141707348>, or unsubscribe <https://github.com/notifications/unsubscribe- auth/AAACPAC3S3AM5HLABHHCASTVMWU55ANCNFSM5XHYPSOQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[elight]

That is, I'm not doing anything that lends itself to tying up an event loop unusually.

…

On May 31, 2022, Evan Light ***@***.***> wrote: I do use dataviewJS but I don't on that particular note, just DV's query language. On May 31, 2022, Vipercat ***@***.***> wrote: > I wonder if all of you who have the lag issue use dataview JS > queries too ? > > Could it be related ? > > dataviewjs const name = dv.current().file.name; > dv.taskList(dv.pages().file.tasks.where(t => !t.completed && > t.text.includes(name))); — Reply to this email directly, view it on GitHub <#697 (comment) 1141707348>, or unsubscribe <https://github.com/notifications/unsubscribe- auth/AAACPAC3S3AM5HLABHHCASTVMWU55ANCNFSM5XHYPSOQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[YannickTemplay]

I just exchange messages with Zsolt, the author of Excalidraw about the issue he had with Dataview’js but it seems unrelated to our problem.

[YannickTemplay]

His problem was that if there is multiple exact same Dataview JS queries in a vault, sometimes it would not update properly.

I definitely have the same Dataview JS query multiple times in my vault and it I don’t have any update problems.

[YannickTemplay]

Thank you for that video. It is very helpful in showing the problem, but not in giving me enough info to be able to address it.

Hello @claremacrae

I am very sorry but I can't reproduce the problem with a clear set of steps. It happens quite regularly but I can't find the exact conditions.

As soon as I find some more useful info I will come back. I will make sure to include the console.

Many thanks for your software anyway. I could not use Obsidian without it...

[claremacrae]

I am very sorry but I can't reproduce the problem with a clear set of steps. It happens quite regularly but I can't find the exact conditions.

Hi @YannickTemplay - absolutely nothing to be sorry about! And thank you very much for trying.

As soon as I find some more useful info I will come back. I will make sure to include the console.

Perfect - thank you.

Many thanks for your software anyway. I could not use Obsidian without it...

Thank you so much for saying so. It's a team effort!

[YannickTemplay]

Hello @claremacrae

I could reproduce the problem and tape it including the console this time...

See -> https://www.loom.com/share/6dda09e7a03546c6b624698f8d8a9707

Does it help?

I am running tasks 1.11.0 on Obsidian v0.15.8

Thanks!

[claremacrae]

@YannickTemplay - thank you!

I'd like to explore whether the daily note page is in fact open but hidden.

• You have the side-panes closed: I don't suppose you have the daily note pinned in either of the side panes? Or anything pinned in the side pane that embeds the daily note page.
• Oh, or maybe you have another window open but minimised, that shows the daily note?

I can't think of any other reasons for it to still be refreshing tasks blocks if the daily note page is not open.

Also, I see that there are two file names being updated - 07-26 and 07-27. So it's two open panes to search for.

If there are not any panes open and showing those two files, the next thing to explore is whether any other plugins are somehow keeping those two files open.

(Tasks doesn't check whether any panes are visible before refreshing them, it just refreshes them. )

[YannickTemplay]

@claremacrae I don't think I have this page open anywhere.

if you don't mind I will send you a full screencam but because it's my private vault I will send you the link on your private email ?

[YannickTemplay]

@claremacrae ah ah !! I think I found what's wrong. I just sent a screen cam to your github email...

[AnnaKornfeldSimpson]

EDIT: Moved to #959
Hope this is a reasonable place to put this:
I just noticed that indexFile in Cache.ts does a cached read of the file and then getTasksFromFileContent splits the entire contents by "\n" even if there are no list items in the file and the rest of the function is therefore a no-op. Opportunity for small optimization?

[claremacrae]

I haven't said this out loud, but my plan for this issue is to create separate issues for each individual optimisation, or potential one, and then close and lock this one, as the question has been answered, and we need the actions to be manageable.

Hope this is a reasonable place to put this:
I just noticed that indexFile in Cache.ts does a cached read of the file and then getTasksFromFileContent splits the entire contents by "\n" even if there are no list items in the file and the rest of the function is therefore a no-op. Opportunity for small optimization?

Good point.

Related to this, I am not sure whether it checks the file extension, or does the above for non-markdown files too.

@AnnaKornfeldSimpson If you have time, please could create a separate issue with those 2 ideas in? Thanks.

[AnnaKornfeldSimpson]

Related to this, I am not sure whether it checks the file extension, or does the above for non-markdown files too.

Obsidian has a dedicated API for just looking at Markdown files and Tasks is using it here, so that is already taken care of.

[goncalomcorreia]

I'm not sure what's causing it, but Obsidian severely lags when typing when Tasks plugin is enabled on my Mac M2. It works perfectly on my Android's phone app, though.

[claremacrae]

I'm not sure what's causing it, but Obsidian severely lags when typing when Tasks plugin is enabled on my Mac M2. It works perfectly on my Android's phone app, though.

@goncalomcorreia Do you use Live Preview?

[goncalomcorreia]

I use it, but I tried to disable it and it lags just the same. In the meantime, I realized that it only lags when I'm typing in the task lines (lines that start with - [ ])

If I type elsewhere, there is no lag at all!

[claremacrae]

A summary of what we know so far, that will eventually get added to the docs (and the performance improved over time)

• Download this file and read the tips: https://github.com/obsidian-tasks-group/obsidian-tasks/files/9639226/891.-.Issue.-.document.performance.tips.md
• If you use Live Preview, read the description in perf: Live Preview keeps tasks blocks open after switching to Source View #1177 - and especially the Workarounds section.

[goncalomcorreia]

Thanks! It seems that something that I need to give up on is autocompletion, though :( I have been using it a lot, but now that my vault is too big it's impossible to use the task plugin that way (being able to use it with the autocompletion feature)

Do you think this will get a fix in the near future? I may have to look for another task management system, otherwise... The performance tips either don't work or remove me from the ability to use autocomplete and that was one of the main features that draw me in :(

Isn't it weird that these problems do not exist on the mobile app? My first intuition would be that the phone's CPU would be less capable of handling this than the Mac's, but the opposite occurs.

[goncalomcorreia]

Would it make it faster if I kept all my tasks in a single file? This was not in the tips list but I was wondering if it could help

[claremacrae]

Would it make it faster if I kept all my tasks in a single file? This was not in the tips list but I was wondering if it could help

No it wouldn't.

[claremacrae]

Hi @goncalomcorreia

Would it make it faster if I kept all my tasks in a single file? This was not in the tips list but I was wondering if it could help

I would like to help you, but we are currently spamming a lot of users here.

Please start a new Q&A issue Discussion to describe your specific situation, and I can try to give you some more specific advice.

[goncalomcorreia]

Thanks! So sorry for spamming, I thought it was related.

[claremacrae]

Thanks! So sorry for spamming,

Sorry, I really didn't mean to say you were spamming - I apologise that it sounded like that.

I thought it was related.

So did I, initially.

But now that it seems that it is associated with autocomplete, I suspect it is different - and I would like to ask you some more questions about your setup... In case you have discovered an un-reported problem.

And because this is a long thread, there are 12 other people subscribed to it, so I feel it would be a bit better to discuss it somewhere where only you and I will get notifications.

Please, do start a Q&A Discussion, if you have the time to chat more about autocomplete performance issues.

[claremacrae]

Excellent - discussion on autocomplete performance moved to #1343 - thank you.

[tonyxiao]

@claremacrae is the performance much better with autocomplete disabled? Also seems that the performance has regressed in the latest 1.8.1 point release, did something change?

[claremacrae]

Not aimed at anyone in particular, but I have been meaning to lock this issue for some time, as it has become too long to be useful.

If anyone comes across this issue and has a problem with the performance of tasks, please:

1. Follow the steps in the links in this comment
2. If you still have a problem, open a new Q&A Discussion, and give information about the specifics of your situation, and report in detail what you have already tried.

Many thanks.

[claremacrae]

Closing, as this has been replaced by separate issues on areas to improve performance.