Tag Usage & Tagging Strategy

[BethanyG]

Ported from Issue #73 -- Please see the prior thread there for more context.

---

Please also review this PR comment as background: #69 (comment)

---

The TL;DR

We are going to be tagging literally everything, and now is probably the time to have a discussion about our thoughts and architecture around it.

Some Background

• We've decided to use a Django add-in library called taggit to manage tagging for our backend API points. We currently use the default configuration.

• Taggits default configuration uses Djangos contenttypes framework and the Django ORMs generic relation to model a manytomany relationship between items to be tagged and tags.

• We can change this behavior to be more direct and more explicit by using what is called a through model, but this requires tracking tags for, say Resources in a resources_tags table, and tags for Hangouts in a hangouts_tags table.

What this means

(not an exhaustive list of points, but a start)

• We can "tag" any model (resources, hangouts, people, discussions, notes) in a relatively straightforward matter by adding a "field" that points to taggits TaggableManager class. See this for basic usage. The TaggableManager takes care of looking up and saving the appropriate entries into the three associated "general" tables used for tagging.

• We can use common taggit code to pull associated tags for an item, update those tags, or remove those tags, without having to write additional ORM code.

• We can use common code to serialize and de-serialize tags for our API.

• Pulling tags across many items or endpoints could get expensive, due to the generic relations used. See this warning from the docs. However, at the anticipated usage level for CodeBuddies, this is probably premature optimization.

• Someone cannot easily trace the relationship between the models being tagged and the tags/taggeditems/contenttyes from looking at the DB - there are no explicit links from, for example, the resources_resource table to the taggit_tageditem table.

• Custom serializer code had to be written to properly serialize the tags into the format we wanted them for for Resources. We'll need to use the same code for any other endpoint that uses tags and that code may still have bugs in it.

Some Questions

(again, not exhaustive, but a start)

• Do we re-configure taggit to use explicit relations for each item being tagged? This would create resources-tags, hangouts-tags, notes-tags, etc. Pros? Cons?

• Do we want a explicit Tags endpoint to handle things like translating, filtering, pulling relations, etc? -- we've already done this

• What makes sense strategy-wise going forward for tags?

• Because of the queries involved, will we be setting ourselves up for performance issues? What types of queries do we want around tags??

• Some additional related concerns in the conversation here: Issues 43 & 67 #69 (comment)

• Some side debate about generic relations here

[BethanyG]

Just putting a comment in here to bump this discussion. Currently, we are thinking through where we flag certain attributes vs tagging things. (see the discussion on #157 for one area, and codebuddies/frontend#137 for another area where tags/tagging is mentioned).

[lpatmo]

Do we re-configure taggit to use explicit relations for each item being tagged? This would create resources-tags, hangouts-tags, notes-tags, etc. Pros? Cons?

Clarifying question: this would involve the creation of new tables for each of those tag relations, right? (The same thing as creating a custom tag in django-taggit?)

[BethanyG]

Yup. We already have tagging_custom_tag -- this would then separate that out into multiple tagging_custom_tag_xxx tables. See detailed digression below.

[lpatmo]

What types of queries do we want around tags??

My assumption right now re: tags in user stories is we want:

• to be able to click on a tag related to a resource, and see a paginated list of other resources with that tag
• to eventually (in the new hangout flow) be able to click on a "javascript" tag and see a paginated list of the "looking for" proposals related to javascript
• to eventually (in the new hangout flow) be able to click on a "python" tag and see a paginated list of the "scheduled public hangouts" related to python

Are there any user stories I'm missing?

[BethanyG]

• Ability (in the new flows) for a user to add/remove tags to Resources or looking for proposals or Hangouts, or other entities where they have permissions to do so.
  (e.g. A user tags a Resource or Hangout as Beginner Friendly. Or a user tags a looking for proposal as Mentor Wanted or Teaching or Collaborative Coding.)
• Ability for a user to add tags that they are interested in (either for learning or teaching) to public and private profiles, so that others can pull up expertise and contact info based on them (if the user chooses to share it).
• Ability to add (request?) additional new (not in the system already) tags and query by them (this is probably not something we want for the MVP)
• Ability to view public notes related to a given tag (not for MVP)
• Ability to view public discussion threads related to a given tag (not for MVP)
• Public/Private tags - so the ability for a user to tag things for their sorting interest and in their native language - not a system wide "generic" tag (definitely NOT for the MVP!!)

ooof. There are probably a lot of others I am missing. I will revisit when I've though about it a bit.

[tgrrr]

We might want tags to be linked to other tags. Potentially with hierarchy, but not necessarily

So: we might want JavaScript (parent) to contain React, Vue, Angular (children)

That way, we would also be able to do:
If you are interested in React, you might also be interested in JavaScript...

We may also want a way to associate tags with what the user is an expert in, vs what they want to learn.

[lpatmo]

Do we want a explicit Tags endpoint to handle things like translating, filtering, pulling relations, etc? -- we've already done this

Double-checking: we currently support tags as a field inside resources, but not as something like api/v1/resources/javascript, right?

[BethanyG]

Correct. But Tagging is it's own app -- it just doesn't currently have an endpoint. Tags are included on models we'd like to apply tags to via the insertion of the following code into the given apps model:

# Allow tags to be used across entities
    # E.g. so we can create composite views showing all entities sharing a common tag
    tags = TaggableManager(through=TaggedItems, manager=CustomTaggableManager, blank=True)

Note that this means everything that is "tagged" in the system is (currently) getting dumped into a tagging_taggeditems table that then links to a tagging_custom_tag table (this is the table holding the tag names and slugs for the entire system).

You then relate which tag from tagging_custom_tag goes with what item by linking a content_type_id (e.g.* Resource, Hangout, Discussion, Note) and object_id (e.g. the specific Resource ID or Hangout ID) to a tag_id (e.g. the id of the slug/name combo). The TaggableManager does this right now for us when we call associated tags per resource.

And while we can use the TaggableManager to do this in each endpoint (pull all resources related to a tag and return them in Resources), it probably makes more sense to write these various queries in a tagging endpoint that can then do additional filtering and aggregating around tags as needed....but of course, that's why we are having this discussion. 😄

So to be clear:

1. We can scope a api/v1/tagging/ endpoint (which can then have filters or queries based on the associated items pulled). We'd do something like everything javascript --> filter by resource contenttype or everything Beginner --> filter by hangouts and resources. I am a little hand-wavy on all the details. We'd want to make sure we weren't creating a bottleneck here, and that everything was clean. We would also probably want proper search, and not just a simple filter, although a filter gets us 80-90% there.

2. We can scope api/v1/resources/javascript, api/v1/resources/java, api/v1/resources/react, api/v1/hangouts/react etc. etc. This feels like way too much. But maybe I'm not thinking about this clearly.

3. We can scope api/v1/resources/tags -- and some additional search functionality to do the tag lookup and resource lookup to return resources for a tag. We'd then wan to do api/v1/hangout/tags and api/v1/proposals/tags or some such.

I'm leaning toward the first one.

[lpatmo]

Thank you for the comprehensive explanation! Are you leaning towards the first one partly because it seems like it would be easier than the alternatives to implement?

🤔 Thinking about the GET request... are you imagining like api/v1/tagging/javascript/resources for resources tagged javascript etc., or would api/v1/tagging/javascript (which returns all resources/hangouts/etc.) related to that tab be more straightforward to implement initially?

[BethanyG]

I'm leaning toward the first scenario I described because I think it gives us the most flexibility in how we deal with tags, since the functionality related to them would be isolated in it's own app. So my thinking is that if we changed how we wanted to associate or look tags up, we'd mostly be changing the tagging app, rather than changing every single other endpoint. But I'd have to mock it out more.

I don't think we should do a pre-set endpoint for different tag values if we can avoid it. It doesn't make much sense to me to have 200+ endpoints (just a guess on how many different tags we could potentially have) just so we can "easily" fetch something. But it has been a bit since I dug into all this - so it may just be a matter of setting things properly in DRF views -- and then having the endpoints "fill" based on URL rules. So we'd want to check the taggit docs on filtering and the DRF docs on viewsets on that, and do some experimenting. In particular, I'd want to play with what taggit.similar_objects() might give us in conjunction with a DRF viewset.

I think I am leaning towards /api/v1/tagging/?search=javascript, with a filter of resources/hangouts/proposals, etc. or, if needed/necessary api/v1/tagging/javascript/?search=resources or api/v1/tagging/resources/?search=javascript. But again, we'd probably need to mock/prototype to really get a bead on it.

[lpatmo]

I think I am leaning towards /api/v1/tagging/?search=javascript, with a filter of resources/hangouts/proposals, etc. or, if needed/necessary api/v1/tagging/javascript/?search=resources or api/v1/tagging/resources/?search=javascript. But again, we'd probably need to mock/prototype to really get a bead on it.

Ah, I really like that. I can see either of these alternatives being fine:
/api/v1/tagging/?search=javascript&type=resources
api/v1/tagging/resources/?search=javascript
api/v1/tagging/javascript/?search=resources

... but I also haven't dug into the details of implementation to know which one is more straightforward.

[BethanyG]

While I am thinking about it, I should also put here that there may be a way to "simply" add tags to a fully-fledged search for any endpoint. But I don't want to promise. I think the last time I looked a doing that, I couldn't really make heads or tails of what needed to happen. But I will see what I can find.

[watchtheblur]

What issues comes up when adding tag searching to any endpoint? For the backend, I've been looking into Datalog and (at least in that domain) the problem is easy-ish. Not sure about exposing APIs though. What would the APIs need to accommodate tag searching?