How to improve quality of newly published beta kata?

[hobovsky]

The topic of the discussion has been bothering all Codewars community for a very long time. Discussions about beta quality and tensions between reviewers and authors are, unfortunately, a thing well known to most of Codewars users. In the past, it happened that new, low quality kata appeared in "waves", with one or a couple of authors publishing them in a short period of time. For some recent time, things have been calmer and quality of betas maybe did not improve, but there were not so many of them incoming. For a last couple of days a bunch of new betas and motivated authors appeared, and the discussions about quality and privileges will most probably reappear, that's why I create this discussion to collect some ideas.

The question is: how to make sure that newly published beta kata are of sufficient quality?

[hobovsky]

Raise Honour requirements for kata authors

Currently, it's very easy to gain authoring privileges, since it requires only 300 Honour to get them. Collecting 300 Honour is not difficult, and it takes not much time to get there. After getting the privilege, new users feel a sudden urge to use it, publishing some kata for their fun and joy of others.
One commonly proposed solution is to raise Honour requirements needed to author new kata. While it sounds logical, I am not sure it will work and solve the problem of quality of new kata. I think this idea causes more problems than it solves.

What would be appropriate requirement?

If the amount gets changed to, let's say, 500, or 2000, or 5000, then users will start publishing bad kata just after reaching the new threshold. Some time ago, together with B4B, we tried to perform some analysis. It was not very scientific and in-depth one, but we collected some data about published, approved, and rejected kata, and it turned out that there was not that much correlation between author's Honour and quality of a kata. A user with just above 400 Honour published a kata which was nice, and after a couple of iterations got approved. On the other hand, a seemingly experienced user with 4k Honour published three kata which got promptly auto-retired. And these two cases were not really ones of their kind. There were low-level newbies with nice ideas, there were low-level users who turned out to be experienced coders, there were high-level, experienced users who couldn't be bothered to put proper amount of effort into authoring, and there were high-level users who just speedrun into 3k Honor and lacked authoring experience. Actually, it was very difficult to find a correlation between Honour and quality. That's why I think that raising the Honor requirements will not work well. Maybe it could be tried and tested, but I am not sure it will significantly reduce amount of tensions between reviewers dissatisfied with quality of kata, and authors dissatisfied with opinions of reviewers.

Do you think raising the level requirement would be good idea? What new level would you propose?

[hobovsky]

Posted by XRFXLP on Gitter:

1. I think the analysis done by you and @Blind4Basics is most probably affected by sampling bias (because number of users with ~500 points > number of users with ~5k honors)
2. Even if point 1 is not true, I still believe that raising honor requirement will improve the beta kata quality hence overall kata improvement statistically (and more so, if the all duplicate approved kata is removed/retired which will decrease the contrast between users who grind on 100 array-search and those who just want to solve good katas)

(and the speed by which de-duplicated kata are being retired, it will take almost 10-15 years (they are just tooo much))

[hobovsky]

Idea: Make privileges for introducing new content into Codewars non-public

What I personally think would help is limiting privileges of regular users to introduce new content into Codewars. I think that ability to create new kata is not the actual problem. I think the problem is that kata are too easy to publish, and to approve.

My idea would be to not modify requirements for authoring, but for publication and approval of new kata. I think that users should be allowed to create new kata, but the kata should not pollute the global pool of content available for others. Kata would be marked by its author as available for review, and then users with (possibly moderated) review abilities would review it and reject or approve for publication. When approved, such kata would be available for beta process as we currently know it.

The honor requirement could be increased too, but I am not sure if it needs to, and to what level.

[JohanWiltink]

Do we force people to participate in Discord to find a reviewer? Not everyone wants to be on Discord, but rigging new infrastructure on Codewars may be quite an investment of somebody's ( cough .. @kazk ) time. Not sure that is or is not worth it .. oh well, implementation detail.

[hobovsky]

Do we force people to participate in Discord to find a reviewer?

No, I don't think a membership on Discord should be any significant factor for any in-site activities. It makes things easier, but should not be a requirement.
Reviewers might organise themselves on Discord, but I think users should be fully capable of achieving their goals just by using the site.

[JohanWiltink]

The whole scheme would, of course, require time from reviewers. We probably have enough people for that, though it might slow down people getting their first kata published.

I don't really have a problem with that - creating kata is not what Codewars is about ( solving kata is, and there are enough of them where we won't have any chicken and egg problems ), and creating kata is of a totally different order of magnitude than solving them.

[Kacarott]

There already exists a privilege for some users to view pending translations, what I imagined when I said this: "Additionally the privilege/ability to review draft kata for publishing could be given to experienced authors" was that authors with X number of approved katas (>5?) would automatically gain the privilege to look at the "queue" of new authors draft katas, awaiting publishing/feedback.

[JohanWiltink]

Still a pull system. The push often comes in the form of a Discourse or Discord request, but ideally there would be some way on Codewars for authors to explicitly request review. ( Translations push notifications to the kata author - this would obviously not work for Draft kata. Then again, it doesn't work for translations on kata from inactive authors either. )

[hobovsky]

Idea: allow only experienced Codewars users to publish new kata

There were ideas to allow publication of new kata only for users who registered X days/months/years ago, or reached Y kyu rank, etc.

The problem I see with this approach is that, just like Honor requirement, it does not measure authoring skills of a user. While experienced users are indeed more probable to know what a good kata should be, it's still difficult to measure actual experience with user's kyu rank or age of account. Additionally, users with new account are perfectly able to be experienced programmers or reasonable authors.

Do you think that some other requirements than Honor could help?

[hobovsky]

Idea: improve documentation on authoring

A couple of posts above, Johan mentioned one thing I forgot to mention before: docs. Maybe improving documentation in some way would help? Maybe what we currently have is bad, insufficient, or difficult to find or follow? Do you think existing docs on authoring are ok? Maybe there is some area which requires to be redone? Maybe something very important is missing? Maybe docs should be done more ubiquitous or discoverable, for example by embedding them (or links) into the site, kata editor, etc?

Will people read docs at all?