[meta] development & user growing pains, users stories that fail right now and why

[jinnatar]

I'll try to collate some meta issues I've started to see and specific things that could make things better:

TL;DR; the core + bots ecosystem isn't a small scale hobby project and in my opinion needs more rigorous development practices & self-control to continue being successful, both from a developers and users (someone just running a bot) point of view. We should probably talk about this and make conscious choices one way or another.

User stories gone bad, my root cause analysis and fix recommendations:
(happy to split these to separate issues if we can agree that they are issues we want to fix)

1. "I pulled the latest Bot but this one part of it fails in weird ways"

   1. The documentation may not be up to date, ergo following the install docs didn't get them to a good state.
      • The only way to really catch this is more rigorous pull request vetting. One way to facilitate this is a template for pull requests that goes through a simple checklist such as "did you add any new dependencies?" .. it's totally OK to have a pull request open for as long as it takes to make it a good citizen in collaboration.
   2. The sample config may not be up to date. Any new variables need to be added there, otherwise the new user will miss them. Pull request scrutiny is a good solution here as well.
   3. An outdated core version is pinned to the bot they're setting up. Git submodules are always tied to a specific revision, the idea being that the developers know which revision is needed, ergo they pin the revision that's needed and everything is at least somewhat manageable to the user. (I'll expand more below why this is sub-optimal in any case.) This only works if the bot repos have the core and any other submodules pinned to a known good revision that has exactly the features expected and no surprises. Supporting both the submodule & separate core installation is tricky here since the two use-cases can get a very different version of the core. Say for example that someone changes a core method that's used by all bots to work better on bot-A but only updates that one bot to take advantage of it. Someone using a pinned version core in bot-B is fine, they have the exact version they need. However someone using a "floating" core that's the latest, or whenever they last updated it, may get the core that's compatible with Bot-A but not Bot-B. To fix their conundrum they need to stop using a single core and maintain multiple until all bots catch up to the latest core.
      • Right now users are literally jumping into a development repository that's in an unknown state and hoping that what they pull is complete and coherent. The only sane solution both for users and developers is to have a dev branch that when stable, will become the next release. When it's time to cut a release it's a good opportunity to make sure that all the pieces fit together and make sense: the right core revision, the right assets, an updated example config and config version and that the whole thing is tested to actually work, not just the parts that the specific developer was using.
   4. There is no testing, not "real" testing anyway. When a project becomes sufficiently large it goes past the point of any one user using all of the functionality themselves. No one person is probably using both webhook raid insertion and manual insertion, ergo any change to one or the other or to something seemingly unrelated may break only one of them, or some issue that is found is only fixed for one or the other.
      • There's two major branches of software testing, feature and unit testing. Feature testing is difficult and time consuming with a large feature set since it means either manually or automated running through an ever increasing set of features and making sure they still function the way you expect. Catching corner cases is hard. Unit testing on the other hand is easier in theory, but only if you thought about it from the get-go. It means identifying small "units" of functionality that can be tested in isolation. Say for example adding the text to the raid image could be a unit and an example unit test would be: "iterate through this list of every Pokemon name ever, render them into the raid image and verify that no black pixel is at the edge of the produced image." .. this test proves that all names fit into the image. If the test was passing previously but now with a change your proposing in a pull request causes that test to fail, probably your change is either bad or more likely caused unintended consequences. All of this can be automated btw in GitHub, all the developers need to do is that whenever they add a "unit" of code, they add corresponding unit tests that confirm the unit is working as intended.

2. "I want to get the shiny new feature, how do I update this thing again?" or "What's this git stuff, can't I just download something and stick it in FTP?"

   1. Since there are no releases, you need to either start from scratch with a new clone, or know quite a bit about git. Doing a git pull at the bot repo only gets you updates for that repo, git is meant for development and only does exactly what you tell it to. You'd need to run a git submodule update separately to also update any submodules to the latest version pinned. This is a simplification, in reality this is really not user friendly: https://stackoverflow.com/questions/1030169/easy-way-to-pull-latest-of-all-git-submodules ergo this should not be a pain that users need to deal with.
      • There should be GitHub releases that provide a working whole, think traditional versions. "I have 1.5 but 1.6 fixes some of the issues I've had, let me update to that one." or "I have 1.6 but 2.0 adds those fancy raid pictures, I should probably update to that one." .. The user can then download the release zip file off GitHub and push that into their FTP space or whatever, no need to deal with Git at all. Or if someone is using an actual unix shell for their hosting then they can pull the latest from the master branch and safely assume that it's a working whole, the latest stable version.
   2. The current version system is a step in the right direction but ultimately doesn't guarantee anything and changes too sporadically to help anything. The sample config seems to perpetually have an out of date version which means even a fresh pull is screaming at you that the config is old. Any automated versioning system is somewhat doomed to fail unless it's tied to releases. Development should not have versions, only releases deserve versions.
      • If semver (Semantic Versioning, recommended by GitHub for all projects: https://semver.org/) is not preferred then the time based version needs to be simpler and more human understandable. Think Ubuntu versions (19.10), ISO-8601 style (2019-12-10) or simplified (20191210). But honestly, semver is the de-facto standard for good reason.

3. "I cloned the repo, I pulled the submodules, but this one feature doesn't work anyway, I might be missing some assets that are not bundled?"

   1. Some assets are bundled with the project, some are external with php scripts for getting them. This assumes some competency from the user that isn't really fair, for example having php-cli installed in the first place to run them. Without the assets some features just plain fail to work. I do understand that there are some copyright issues here and the asset repo is quite large which also makes it somewhat suboptimal to embed as a submodule.
      • One potential way to make this more user friendly would be to use assets that are not copyright burdened allowing easier embedding instead of shifting the burden to the user. There are also open RESTful APIs for Pokemon data, including sprite data. The downside is that the default pictures wouldn't match what people see in the game which is a bit of a bummer, but at least it wouldn't be in the "grey zone". The current scripts could work as a fallback if someone really wants to go get the official assets at their own risk.