[SPIKE] Use Swiftype for search

[chillu]

Overview

The current search is useless: It indexes changelogs, which leads to a large amount of irrelevant results. It also uses a one-off search solution (Lucene PHP), which is hard to maintain and customise. While we could fix the short-term issues, there's little value in keeping a custom search engine overall - particularly now that we have Swiftype across silverstripe.com and silverstripe.org already.

Acceptance Criteria

• Crawls doc.silverstripe.org and api.silverstripe.org - WHY API? CHECK WITH INGO
• Search indexes supported major and minor versions (3.5, 3.6, 4.0, 4.1) - CHECK WITH INGO
• Search indexes supported master
• Search defaults to currently viewed version
• New content is automatically picked up in search results (after max 24h)
• Documentation search results are integrated on silverstripe.org main search
• Search results show extract
• Search results show the source domain
• Search results exclude changelogs
• Search queries get picked up by Google Analytics
• Prioritises results from https://www.silverstripe.org/learn/lessons/ - GLOBAL SEARCH OR SITE SEARCH?
• You can link to search results
• Any code, templates and documentation for previous PHP-based search are removed from the doc.silverstripe.org codebase
• Works with code searches, ideally without quoting the terms (e.g. versioned_gridfield_extensions, SilverStripe\Security\Member or DB::query)
• Uses the same swiftype account as silverstripe.org & silverstripe.com

Notes

• We should just remove old versions from doc.silverstripe.org, rather than doing any form of whitelisting in Swiftype - which will be a burden every time we bring out a new release.
• Does not need custom styling, just use whatever Swiftype does by default
• There are 414 *.md files in docs/en for 4.x (242 of those are changelogs)
• There are 760 matches for api.silverstripe.org/4/*
• Overall we're looking at less than 5k docs.
• Currently the docs are building from branches rather than stable versions, showing docs for unreleased feature - which is confusing. That's out of scope here.
• No need for explicit search filters for versions, just search whatever is displayed
• We might be able to remove this again if we switch the whole doc.silverstripe.org to ReadTheDocs.org: RFC: Move to using an external documentation builder like readthedocs.org #157
• Swiftype cost approved by Sam

[sminnee]

FYI: "Uses the same swiftype account as silverstripe.org & silverstripe.com" should probably added as a specific AC.

If this winds up polluting search results between main site search and docs search, IMO that's fine, as the distinction between the two is more relevant to us than our users tbh. ;-)

[chillu]

Added :)

[robbieaverill]

Related: silverstripe/api.silverstripe.org#73

[chillu]

Oh here's a good example: Search for versioned_gridfield_extensions. Ten pages of changelogs, the actual page containing this isn't even in the results: https://docs.silverstripe.org/en/4/developer_guides/model/versioning/#versioned-gridfield-extension

https://docs.silverstripe.org/en/4/results?q=versioned_gridfield_extensions&Versions=4&action_results=Search&start=0

[clarkepaul]

Related https://github.com/silverstripeltd/cc-issues/issues/135

[sminnee]

As a test, I've enabled docs.silverstripe.org in swiftype with:

• /en/3 and /en/4 as a URL whitelists.
• /en/2/changelogs/ and /en/4/changelogs/ as URL blacklists.

[robbieaverill]

We're relying on /en/3/changelogs for the SS3 changelogs and /en/4/changelogs for the SS4 changelogs though right? Relevant: silverstripe/silverstripe-framework#8035

[sminnee]

I blacked listed them due to this AC. We can always tweak later:

Search results exclude changelogs

[robbieaverill]

Ha, right

[sminnee]

Swiftype has only managed to index 2 page (en/3/all and en/4/all). I suspect that this may be related. silverstripe/silverstripe-docsviewer#148

[sminnee]

OK links are now root-relative on docs.silverstripe.org. We'll check back in a day to see if it can index more than 4 pages.

[sminnee]

That didn't fix it, Swiftype supported suggested that this might be causing issues #180.

[robbieaverill]

Note that @lexakami has just rolled out SwiftType to silverstripe.org, docs, userhelp, and addons.

I don't think those search facets would include each respective site's content, however, but at least the buttons isn't broken any more.

I've closed all the other issues in favour of this one, so we can track any further changes required here.

[unclecheese]

Closing as we went with Algolia docsearch.