How to ensure consistency of a kata between translations?

[hobovsky]

Quite common problem with translations, affecting mostly kata with many available languages (but not only), is consistency of quality and difficulty across translations: some available languages are curated better than others, have more tests, tighter tests, or the other way round: miss some tests for edge cases or some types of inputs. Sometimes translations differ so much that it's even difficult to tell whether they cover the same problem. Of course, it is not possible to make all languages identical, because of differences in performance, paradigms, etc. but except of performance, there is a lot of other aspects which can be unified: amount of test cases, ranges of inputs, coverage of input scenarios, practices related to description, sample tests, edge cases, etc.

The problem of inconsistencies bothers me for a couple of practical reasons:

• Creating a good new translation is difficult because it's not easy to find out what rules should be followed, especially if there are many translations available which wildly differ or with mutually exclusive approaches. Even if description is as complete as possible, not all details can be revealed in the description, and things which are not can be missed or forgotten.
• Reviewing and approving a pending translation is difficult because it's difficult to tell how complete it is.
• Deduplicating kata is difficult, because it's difficult to establish similarity between kata, and pick a good one.
• Fixing issues is difficult, because it's difficult to tell to what practice should the flawed kata conform.

Would you have any ideas how to enforce consistency of kata between languages?

[hobovsky]

My idea would be provide additional area in kata editor, where authors would put their idea of technical requirements for the kata and tests. The field would be similar to description: potentially large, shared across languages, but not visible for users who havent solved the kata (ideally, visible only in kata editor, and kumite view, and kumite editor). When a new language would to be added, reviewers would be responsible for verification whether the translation covers all requirements listed in the snippet. It could also contain some notes on exceptions for some languages which cannot conform to requirements because they differ greatly in paradigms, avaialble libraries, accepted idioms and practices, technical possibilities, etc.

Example notes in the snippet could be (just example points, not related one to each other):

- BigInt must be either not available, or disabled.
- Solutions worse than O(n log n) should not pass.
- Fixed test for 0 must be present.
- Sample tests should contain at least one large input.
- Random tests must be guaranteed to generate some perfect squares.
- This kata does not allow mutation of input, and tests must verify whether the input was not modified by user solution.
- Input string should contain only lowercase letters, uppercase letters, spaces, and  
  punctuation: `.` (period), `,` (comma), `?` question mark, `!` exclamation mark. In the expected answer, words must be capitalized: first letter uppercase, remaining letter lowercase.  
  There should be no uppercase letters in a word expect their initial letter. There should be no adjacent, trailing, or leading spaces.

Initially I thought that such points could be put in a comment in a kata description, but I think the comments would be still visible in page source, and sometimes could contain spoilers.
One potential problem with the additional snippet is that it would increase risk of merge conflicts if it would be modified by some translation.

[Blind4Basics]

I like the general idea and your present suggestions. On the other hand, I disagree about making the number of tests consistent across languages. Or at least, not for performances oriented kata. This is too much specific while some languages behave way too much differently to be able to perform the same number of tests. Sometimes you just cannot build the perf tests the same way.

But generic ideas like you showed there, yeah, sounds like a good idea.

About the "where", I think it should be accessible in a similar way to the tests. Meaning it should be visible once the kata is solved, like you suggested, but from the solutions+discourses pages as well. This would need to be accessible to the beta testers since that's part of the contract of the kata.
About that... I'm not really sure how this would "behave" with new authors and the beta process. The cost might become higher to get that correct as well as what we already are looking for.

[Kacarott]

I quite like this idea. At first I thought "But this is what the full tests already do, you just have to read them carefully when translating", however not all translations are faithful, and people dont always use the original language as reference when translating, resulting in kata where almost all translations are not following the original idea.

This idea could probably actually be combined with #2588 . In the same document, authors can describe the intent, etc. along with a rubric that translations should follow (like in your example) since the rubric is also a description of intent, in a way.

[hobovsky]

I like the general idea and your present suggestions. On the other hand, I disagree about making the number of tests consistent across languages. Or at least, not for performances oriented kata. This is too much specific while some languages behave way too much differently to be able to perform the same number of tests. Sometimes you just cannot build the perf tests the same way.

I agree that performance related things are the most difficult to be done consistently due to differences between platforms, and that's why I tried not to put too much stress on performance tests in the above idea.. Still, the note could contain something very general, like "tests must contain cases with squares of primes", or "expected time complexity is O(1)", or "gauge difficulty of tests by manipulating length of input strings", etc.
But I probably was not clear enough, because by "number of tests" I did not mean specifically performance tests, but rather situations when some relatively simple kata contains 50 tests spanning over its whole input domain in one language, and 2000 tests spraying RNG all over in hope of getting a good coverage in another language.

This idea could probably actually be combined with #2588 .

Yes, the #2588 could be one solution, but I asked a more general question here because maybe users have some other ideas of tackling inconsistencies, other than an additional editor field.

About the "where", I think it should be accessible in a similar way to the tests. Meaning it should be visible once the kata is solved, like you suggested, but from the solutions+discourses pages as well. This would need to be accessible to the beta testers since that's part of the contract of the kata.

I am not sure I understand, a contract between who and who? A contract between a task and a solver should be ideally presented in description (and in some special cases in solution setup, and/or exceptionally in sample tests). A contract between a kata and translators needs not to be visible to solvers and I think it would be enough to present it in areas related to authoring. But you make a good point of betas, because when reviewing a beta, reviewers do not immediately access editor, so they couldn't see the snippet easily. They would have to create a fork to review it.

I'm not really sure how this would "behave" with new authors and the beta process. The cost might become higher to get that correct as well as what we already are looking for.

This is a good point. Making people to learn how to fill in the snippet might be not easy.

[Blind4Basics]

a contract between who and who?

Not sure how to put that into words... I was thinking about this like "an underlying contract the author would like to enforce on the kata/solver". Something like an abstract description of the wanted requirements. The solver doesn't need to see it, but reviewers need to see it to be able to know if the implementation matches the intents.

Not sure if I'm clearer...?

[hobovsky]

Yup, got it, and it's a fair point: reviewers mostly use only the kata trainer. I personally fork betas to review code, but I am not sure other reviewers do. I also never enter the editor.

[Kacarott]

Another idea which would likely help with this a lot is #2382 . More eyes on a translation, and "waiting period" while it collects a couple solves would likely help spot problems in translations a lot more consistently.

[hobovsky]

Another idea which would likely help with this a lot is #2382 . More eyes on a translation, and "waiting period" while it collects a couple solves would likely help spot problems in translations a lot more consistently.

Two things which I specifically wanted to target with this idea are:

• Things expressed with the code in an indirect way, which can be difficult to spot when reading code,
• Inconsistencies not only in incoming kata and incoming translations, but also in existing kata. The idea you refer to is a very good one, but helps only with new things. On the other hand, one of my ideas here is to take some specific kata, analyze requirements of its task, write them down in the snippet, and then go through all approved languages of the kata and make them conformant to the requirements. Having such a summary would make fixing kata easier.

[seanburlington]

I've been refreshing my SQL recently by doing some kata and this inconsistency is a real problem

It's had to find good SQL kata because they don't translate well and instead of practising WHERE, JOIN and GROUP - which is normal for SQL

Instead kata focus on things like functions - which are (in my experience) unusual in SQL

In fact I think a beginner could pick up some really bad habits

https://www.codewars.com/kata/5694cb0ec554589633000036/sql

You could make it a different level kata but it's not really a hard problem either - it just uses a relatively obscure piece of functionality.

I think some things don't translate and should just be removed

[awesomeAD1]

I like the idea. For super duper convenience, the requirements set out by the author could be translated into a literal checklist available in the translation/fork, where the translation author checks each box to signify to reviewers that this particular aspect of the spec was considered and implemented.

On a side note, related to the issue of "translations of translations of the original language": I agree this is a lot like the Telephone Game, where each "copy" gets progressively worse/off spec. Just today I had to look up Python tests to see if a Rust translation was following the same spec. However, I have no way of knowing whether Python was the original language, and might itself be off-spec. A hint on the details page of a kata as to what the original language was might be helpful.

[hobovsky]

I think that for majority of kata knowing the original language is not helpful at all, because original languages tend to be highly specific rather than generic, and requirements include things which do not translate well. One symptom of this problem are kata which use a list of Object in a statically typed language just because it was originally a heterogenous list in the initial Python or JS version and C# or Java translator translated it mindlessly into List<object> or List<T>. Or kata which return Object because sometimes it's an array, and sometimes a string on failure.

Knowing initial language would be helpful if it could be assumed that the author of the initial version knew what they're doing.

Side note: list of languages in the "kata stats" section is presented in the order of merging, or order of publication. So the first language on the list is the initial language.