Some thoughts on API access to Tasks features

[claremacrae]

Tasks API integration - overview

This is a brain dump - and notes dump - of @claremacrae's notes on possible Tasks plugin API/facilities for integration with other plugins.

• Saved in Some thoughts on API access to Tasks features · Discussion #1646 · obsidian-tasks-group/obsidian-tasks · GitHub

1 Existing Requests

• Add a way for third-party plugins to parse and modify Tasks #1389
• Provide API for developers of other plugins #913
• Expose the renderer classes for use outside of the context of the MarkdownPostProcessor #361
• How to extend Tasks? #1245

2 What I feel Tasks offers

• Recurrence
• The search language

3 Plugins interested in integration

I've been keeping a canvas about the requests received...

Here is the source canvas file, for working links:

Tasks - Helping other plugin developers - version 1.canvas.txt - remove .txt extension to use

4 Accessing the code - npm or this.app.plugins.getPlugin

There's been a lot of discussion about whether to:

1. follow the dataview route of offering an npm package that plugin developers can use, so that their users do not need to install the Tasks plugin.
2. stick with exposing an API that plugin developers and users can call via this.app.plugins.getPlugin("obsidian-tasks-plugin")

The conclusion is to start with the this.app.plugins.getPlugin("obsidian-tasks-plugin") route.

5 Existing integration code - and examples

5.1 In main vault

5.1.1 Render methods exposed - undocumented

• Created in Expose render methods by nothingislost · Pull Request #363 · obsidian-tasks-group/obsidian-tasks · GitHub

Example code:

const code = "due after 2021-10-20"
const el = document.createElement("div");
const ctx = new Component();
const renderer = this.app.plugins.getPlugin("obsidian-tasks-plugin").queryRenderer.renderer
renderer(code, el, ctx);

5.1.2 Access to all the current tasks - undocumented

• Public readonly-access to all the tasks loaded from the vault.
• Must not be accessed until the cache is created and populated.

obsidian-tasks/main.ts at bf2ac0fb5a66abd899b477686d45c6e3969d4aa5 · obsidian-tasks-group/obsidian-tasks · GitHub

5.2 In Clare Macrae's fork

5.2.1 Running Tasks queries and accessing results

• Added a search() method on plugin.
• It returns the tasks grouped, which can be a little hard to iterator over

obsidian-tasks/main.ts at c302c57ae4a5ad2f1aa2d071c3d8565f24229e12 · claremacrae/obsidian-tasks · GitHub

Example of use:

```dataviewjs  
const tasksPlugin = this.app.plugins.getPlugin("obsidian-tasks-plugin")  

// Return the number of matching tasks, as a comma-separated string
function getQueryTasksCount(source) {
   return tasksPlugin.search(source).totalTasksCount().toLocaleString();
}

const today = new Date().toISOString().split('T')[0]
const total = getQueryTasksCount('');
const open = getQueryTasksCount('not done');
const done = getQueryTasksCount('done');

dv.el('pre', `| ${today} |             ${total} |             ${open} |            ${done} |                         n,nnn |`)
```

5.3 Related PRs

5.3.1 Group by functions #1421 - not merged

• feat: Group by functions by weirdhorror · Pull Request #1421 · obsidian-tasks-group/obsidian-tasks · GitHub
• Proposal to allow users to supply JS expressions for custom grouping

5.3.2 Add hooks to let other plugins extend the Tasks plugin. #1249 - not merged

• Add hooks to let other plugins extend the Tasks plugin. by alexthewilde · Pull Request #1249 · obsidian-tasks-group/obsidian-tasks · GitHub
• Add hooks to let other plugins extend the Tasks plugin.

6 Backwards Compatibility and other concerns

6.1 Some personal wishes

• In order to try to minimise user support time on Tasks, I am very keen to not release features that may cause breakages for users in future
  • So any documented API would need some confidence that we could maintain it for a long time
• There are several major missing abstractions in the Tasks code, like no types representing dates, the global filter, parsing of tasks, writing of tasks
  • Since taking responsibility for Tasks, I've been learning about the code and gradually making major refactorings
  • So it would really help for any published API to still allow updating of the internal structure of the code

6.2 Some ideas

At the moment, all the sample code above calls methods on the plugin directly.

6.2.1 Versioned API

I was thinking instead it might be good to design for future updates, so something like:

```dataviewjs
const tasksPlugin = this.app.plugins.getPlugin("obsidian-tasks-plugin")
const api = tasksPlugin.apiV1()
const tasks = api.search(source).totalTasksCount().toLocaleString();
```

The idea here is that we publish an apiV1 interface - which would use no Obsidian graphical classes, and so would be very amenable to TDD and automated testing.

6.2.2 No writing of files in the first release

There is a very rare bug that Tasks can toggle the wrong line in a file. It is likely that the Tasks code for parsing and writing Markdown code may be replaced by datacore code, which is more performant and (I believe) better tested.

Until that happens, I would prefer the initial release of a Tasks API to support read-only access to user vaults.

[BluBloos]

currently, I see that the query function will return a Task[].

This is an internal data structure.

Maybe it would be better to return some sort of TaskExternal[] and have a conversion step before releasing the task results from the query?

Could do something like,

class TaskExternal {
  public readonly isDone: Boolean;
  public readonly priority: number;                 // 1 is the highest priority, any larger number is a lower priority.
  public readonly tags: string[];                      // a list of tags, distilled from the description.
  public readonly originalMarkdown: string;  // the original markdown task.
  public readonly description: string;             // the description of the task.
  public readonly startDate: Moment | null;
  public readonly scheduledDate: Moment | null;
  public readonly dueDate: Moment | null;
  public readonly doneDate: Moment | null;
  // TODO: recurrence
}

[BluBloos]

I also have another suggestion:

instead of a one-time query, Tasks could allow other plugins to register a callback.
it works like this:

• at the time of registering, they provide the query string.
• just as any tasks block would receive updates to re-render, the callback will get called with a new list of tasks.

[claremacrae]

First let me say that my development background is in C++ for desktop scientific software applications.

So with this kind of discussion, I am firmly in the "I don't know what I don't know" camp - and very much hoping to be guided by others as to what is feasible and what is worthwhile!

class TaskExternal {

I like the principle of that. It would satisfy my concern of not wanting to expose things I would want to refactor in future...

I started work on introducing a TaskDate class - for #1104. It's on hold a the moment, but we could also use it for the public tasks:

public readonly startDate: TaskDate;

For a better experience for API users, we could always rename:

• Task -> TaskInternal
• TaskExternal -> Task

instead of a one-time query, Tasks could allow other plugins to register a callback.

Wow, sounds interesting.

[BluBloos]

First let me say that my development background is in C++ for desktop scientific software applications.

The bulk of the code I have written is in C/C++ as well :)

So with this kind of discussion, I am firmly in the "I don't know what I don't know" camp - and very much hoping to be guided by others as to what is feasible and what is worthwhile!

I certainly hope other contributors join this discussion as well in case my ideas aren't the best!

For a better experience for API users, we could always rename:

Task -> TaskInternal
TaskExternal -> Task

I agree

we could also use it (TaskDate) for the public tasks

I'll take a look at TaskDate when I get a chance

[claremacrae]

I'll take a look at TaskDate when I get a chance

Thank you.

main...refactor-add-TaskDate-class

[BluBloos]

There is a very rare bug that Tasks can toggle the wrong line in a file.

Is there an issue for this? I'm curious to check it out ...

[BluBloos]

#688?

[claremacrae]

The label is scope: data integrity:

There are issues but no discussions:

https://github.com/obsidian-tasks-group/obsidian-tasks/issues?q=is%3Aopen+is%3Aissue+label%3A%22scope%3A+data+integrity%22

[BluBloos]

re: point 6.2.2, if someone ideates a solution for write-access whilst considering that there will be major changes to replace with datacore code, would that work?

I was thinking about how end users (other plugin devs) would even desire write-access, i.e. what would be useful/convenient.

and I think to an extent, the design of read-access seems related to write-access, and thinking at the same time about how both should work might be useful?

[claremacrae]

I'm sorry, I've read this several times and I don't really understand the questions.

[claremacrae]

If you are thinking of designing what write-access would look like, and feel that its behaviour could be implemented using either the current Tasks code or the future datacore, yes, that would be absolutely fine.

Also, if the proposed safety check were added to the Tasks code, to prevent overwriting the wrong line, I think I would be OK with releasing an API before before datacore had been (maybe) adopted.

[BluBloos]

Sorry for not writing this in a more clear way ... but looks like it wasn't that awful as yes, I was thinking of designing what write-access would look like.

[claremacrae]

Sorry for not writing this in a more clear way ... but looks like it wasn't that awful as yes, I was thinking of designing what write-access would look like.

OK.

[claremacrae]

I found another request: added in the list above:
#1245

[BluBloos]

The idea here is that we publish an apiV1 interface - which would use no Obsidian graphical classes.

I think this is my lack of knowledge here on Ob things ... what are "Obsidian graphical classes"?

[claremacrae]

Yeah, sorry for my haste.

I don't immediately have a single link to point you to, but most of the Obsidian exports (Plugin etc) cannot be used outside of an Obsidian process. Essentially, Obsidian wasn't designed for automated testing.

A bunch of plugin authors have tried working around this.

I am keen that as much as possible any API that is published can be thoroughly tested, automatically. Making everything accessible via Plugin will, as far as I know, make that a lot harder.

(I mean, I know we could have 2 layers, where the function in Plugin was a thin wrapper around something elsewhere, but that still adds overhead when adding new features to the API)

I hope that is a bit clearer.

[BluBloos]

A bunch of plugin authors have tried working around this.

Sounds like you have looked into this? If you have any links that would be greatly appreciated!

[claremacrae]

Sounds like you have looked into this? If you have any links that would be greatly appreciated!

Sadly, no links...

I've spoken to several plugin authors over the last few months.

Most recently blacksmithgu told me that dataview checks line content before overwriting lines. I haven't looked at the code, and don't know the behaviour - whether it gives a warning.

In an earlier conversation with another author, they told me that they just checked if lines matched, and if not, they silently did nothing, that is, they didn't overwrite the file.

Looking at the notes in https://user-images.githubusercontent.com/4840096/217628522-fa2ba1e5-f6cb-45fd-8ede-e018a57400f3.png - it was probably the cardboard author, but I don't having anything in my notes.

[BluBloos]

I'm a little confused by your reply ...

it seems like you've mentioned what other plugin authors are doing for data integrity,

yet i'm asking about what other plugin authors are doing for testing their plugins public API interface?

[claremacrae]

I'm a little confused by your reply ...

it seems like you've mentioned what other plugin authors are doing for data integrity,

yet i'm asking about what other plugin authors are doing for testing their plugins public API interface?

aaagggghhhh!!! 😄

So I've been feeling for a few days that conversation via GitHub issues and discussions is a terribly ineffective and inefficient thing at the best of times - low-bandwidth and too much scope for misunderstanding.

I'm getting 20 to 50 emails a day about tasks, often with no earlier context, and often my browser does not jump to the actual comment...

And when someone is as knowledgable, keen and helpful as you are, @BluBloos, this compounds, as I have to remember which of the several things we have been talking about...

I (hope) I've sent you a friend request on Discord to see if we can arrange a time to chat 🤣

[claremacrae]

Related: from the Tasks 2.0.0 release note:

🌟 Added new public API to retrieve markdown string for a new task via the 'Add or edit task' modal - thanks @3bit
Authors of other plugins can now use the Tasks modal to prompt users for data for a new task.
Feedback appreciated on this feature!
📖 https://obsidian-tasks-group.github.io/obsidian-tasks/advanced/api/

[beesinblenders]

Hello,
This is very cool and helpful!
I'm looking for a way to pre-fill some fields in the modal. Can that be done somehow with the current state of the plug-in?
I mean something like: call this function from within QuickAdd and fill the description field with a tag I'll have passed.
Personally I would use this to add tasks related to projects from these projects' dashboards.
I've looked for a feature request but didn't find one. But I'm not confident that it can't already been done somehow.

[claremacrae]

Please upvote this:

• Support editing existing tasks from the API #1945

(so far mine is the only vault, so it's a long way down the list...)

[ElectronicRU]

Hey,
I think API versioning/TaskDate experiment are a bit of an over-engineering. It's not a production environment, users can downgrade breaking changes at will, etc. I feel like the need to scope out API surface, understandable for someone who has most of their experience in enterprise development (much like myself), is severely bogging this ticket down.

I think this needs to be approached like scripting - moving and letting the user do something is a lot more important than maintaining perfect forward compatibility foresight.

Dataview, for example, doesn't care at all and just gives you the internal types, they're nice enough. It additionally exposes some utility functions like dv.date and dv.duration in order to re-export luxon objects it is built upon. I think this is a good design. Another approach would be to try and support using Date or somesuch at API surfaces, but that's probably harder than just exposing Moment constructors.

If there's interest in building a beta-version of the API that would expose querying (I'm mainly interested in FullCalendar integration so this is the most important feature for me), then I'm totally game. Would appreciate permission to nag over chat, though.

[claremacrae]

@ElectronicRU

1. I've chosen to address some of the opinions you expressed.
2. I am also interested in any code-based ideas you have on exposing an API for Tasks searches.

I think API versioning/TaskDate experiment are a bit of an over-engineering. It's not a production environment, users can downgrade breaking changes at will, etc.

Things you may not have considered:

• When things break, it takes up my time dealing with user disgruntled users. I choose to protect my time, and that of my users.
• You may not call it a production environment, but there are Tasks users tracking things like medication and other important things, and I do feel a certain responsibility to ensure that it is reliable. It's not perfect, but I choose to spend my time on its quality as best I can.
• If a user has two different plugins installed that both use the Tasks API - and one of those plugins uses a different version of the Tasks API to the other, there is no single Tasks version that the user can install to satisfy both plugins... This is one of a number of reasons why the API versioning matters.

Since I wrote this discussion (not issue), I have actually exposed and documented a number of the fields on Task objects, for use in filter by function and group by function.

See Task Properties - and also Query Properties

It turned out that I was able to expose dates in tasks via TaskDate without actually needing to refactor the whole plugin... so to that extent, no, TaskDate has not at all bogged this down.

is severely bogging this ticket down.

Of course you are entitled to your opinion, but I'm guessing you may not have the experience of being the maintainer of a moderately popular open source repo that takes up a lot of your free time - and the resultant juggling of priorities and your own time that is required, depending on when kind volunteers come forward with Pull Requests...

An API for searching does not exist because implementing an API for Tasks searches has not reached the top of my long todo list - and nobody else has volunteered a PR for it either...

If there's interest in building a beta-version of the API that would expose querying (I'm mainly interested in FullCalendar integration so this is the most important feature for me), then I'm totally game.

Yes, I'm interested in what you have in mind.

In case you haven't seen it, there's a lot of help about developing for Tasks in the Contributing guide.

Would appreciate permission to nag over chat, though.

Please explain what you mean by that. Thank you.

[aigoncharov]

I am wondering if we could allow for reactive behavior on task updates.

export interface TasksApiV1 {
    // New stuff
    subscribe(handler: (event: string, data: any) => void): void

    // Old stuff
    createTaskLineModal(): Promise<string>;
    executeToggleTaskDoneCommand: (line: string, path: string) => string;
}

With the new subscibe call we could allow for third-party plugins to subscribe the lifecyle events of the plugin. For starters, we could support a single event taskToggle and pass the associated Task to the handler.

This would allow other plugins to add custom behavior for this plugin. Potentially, it removes the need for onCompletition callback in the core repo.

Also, any other plugin could store its state in tags.

const api = app.plugins.plugins['obsidian-tasks-plugin'].apiV1

// onCompletition support could become a plugin of its own
api.subscribe('taskToggle', (task: Task) => {
    somehowDeteleTheLineFromMarkdown(task)
})

WDYT?

[claremacrae]

I think this may be a duplicate of a comment in a recent PR - if you haven't already done so, could you edit the original so that it directs future discussion here, to try to avoid duplication/lost info...

WDYT?

With so much going on in Tasks, unfortunately I don't have the capacity to consider this right now. (This is why we have started asking for discussion before PRs...)

[LennardZuendorf]

Hey! I’m loving by the advanced querying functionalities offered by the plugin, and I believe it would be highly beneficial to expose these capabilities via an API for other developers. However personally, as I'm building a UI that relies heavily on the core functionalities of this plugin, the more significant pain point is implementing CRUD operations and tracking updates on tasks in Markdown files.

If these functionalities were accessible, I could handle the querying (e.g., filtering by status) on my end, given a structured list of tasks. Essentially, I’d use obsidian-tasks as an intermediary between my plugin and the vault.

Why CRUD?
Implementing code for getting/updating/deleting tasks feels especially unnecessary (I know, being a bit selfish here) since there's already and established an efficient process to retrieve tasks from Markdown, parse them into a structured format with extra fields, and enable updates/additions. Exposing these operations in a structured API would leverage core functionality that’s likely stable over time (even if underlying code may evolve). And given that the task type also shouldn't don’t vary widely, the implementation could be relatively straightforward.

Example:
For instance, there's already have Dataview’s function for retrieving tasks across the vault, like this.dv.pages("").file.tasks. I currently use it (via the npm package) as follows:

const dv = getAPI();
const allTasks = dv.pages("").file.tasks;

// -> Then I have to iterate over each and hydrate it into my internal type based on the text and available key-value pairs in the object.

// Currently I do mapping like so:
const idMatch = dvTask.id ? dvTask.id : dvTask.text.match(/\[id:: ([^\]]+)\]/);
const dependsOnMatch = dvTask.text.match(/\[dependsOn:: ([^\]]+)\]/);
//...
 

With tasks you could provide an easier approach:

const api = app.plugins.plugins['obsidian-tasks-plugin'].apiV1

// getting all tasks
const allTasks  = api.getTasks();

// -> Here the other plugin could either just copy the type of the returned tasks or hydrate again. 
// Either would be easier and more structured than doing a bunch of .match on the task text.

// editing a task
api.editTask({task});


// deleting a task
api.deleteTask({task});

//and so on

TLDR
In essence, exposing basic get, add, update, delete, and watch operations could offer major utility with a relatively straightforward setup. If you’re open to it, I’d be happy to try and come up with a PR that outlines these operations.

Plus potentially looking into providing the task type as well, to make it typesafe. Which I assume is the main benefit of having a npm package, I have no clue how to export a type otherwise....

[claremacrae]

Hi @LennardZuendorf, I understand why you are asking for this... As you will see from the searches linked in the comment below it overlaps with some of the earlier discussions...

This may well happen at some point, but is definitely not going to be implemented by me in the foreseeable future, as there are so many other higher priority requests.

Also, it's a very large responsibility exposing facilities that offer to edit user vaults like this - much greater than editing a single task. I already know that this needs a lot more care than most users presume, due to things like vaults with out-of-date Obsidian caches.

In case anyone were considering offering a pull request, due to the amount of time required to review large PRs, the implementation and testing approach taken would need to be discussed and agreed in advance, for a PR to be reviewed and considered for merging.

In the meantime, the best approach is the use the existing dataview API.

[LennardZuendorf]

Hi, thank you for the quick reply! The request definitely stems from my personal challenges and the amount of “workaround” code I had to write to make the mapping functional.

I understand why this is not be a high priority (probably would be for me either from your perspective). I absolutely respect the amount of work involved in developing and maintaining the plugin, especially with the added responsibility of supporting editing functions and the responsibility of editing user vaults. Admittedly, that part is outside my current knowledge, as I’m just beginning to dive into Obsidian plugin development.

Out of curiosity, what would the process look like for agreeing on an implementation and testing beforehand? Just opening an issue outlining the idea, implementation details and testing approach? I think I’ll keep working with Dataview and manual edits for now, but I’d love to revisit this idea and contribute once I have a better grasp on the development process and finer details.

... I'm probably going to do a hybrid for now and use a combination of dataview api, the functions the obsidian-tasks api already provides and some of my own code for editing.

[claremacrae]

Out of curiosity, what would the process look like for agreeing on an implementation and testing beforehand?

Off the top of my head, it would start with a video call to cover things like:

• Checking that the person had reviewed the Tasks Contributing guide
• ...and had a good level of understanding and experience of test-first development
• ... and of safe refactoring.
• Then we would explore what they intended the user experience to look like.
• And I would also want to explore how the work would be divided up in to a series of small, safe PRs that would be quick and easy to review and merge.

For a couple of years I would have done all this via pairing on the development together. It was great for sharing knowledge - two way - but then realised that I was never doing any actually development myself of the areas I wanted to work on.

[claremacrae]

Existing topics with the label "scope: for plugin developers":

• Issues
• Discussions