Native way to validate crystal docs code examples

[j8r]

Thanks to @maiha , there is already https://github.com/maiha/crystal-examples to validate our code blocks examples put in the comments.
I think it will be useful to have this kind of tool natively in the crystal docs tool.

---

I already have a (very) basic POC, which adds like ~30 line of codes. Some snippets like this one are failing, obviously:

[] of Int32 # same as Array(Int32)
[]          # syntax error

Result

Invalid code block in Array(T):
syntax error in :2
Error: for empty arrays use '[] of ElementType'

Note: the # => results are not checked (for now).

[straight-shoota]

This has been mentioned in previous discussions (#6315 (comment) ff.) and I've had this in the back of my head for some time.

It would be really great to have this!

However, I've been thinking a bit wider: Validating Crystal code examples is a most important feature. But actually, I'd also like to be able to validate code blocks in other languages (YAML or HTML for example). Maybe also general Markdown syntax or a style guide on body copy.
So, ideally a more holistic approach would make much sense. And it could easily be delegated outside the compiler. Just pipe each doc comment to a program that runs various checks for different code blocks etc. Example code checker would be a plugable validator (again based on the compiler).

This approach would have a huge benefit over an internal compiler feature: It can easily re-purposed for other sources of documentation content. The language reference would also get example code validation with exactly the same mechanism.

Now the biggest problem is that to my knowledge such kind of validator multiplexer doesn't exist yet. At least not that I'm aware of. There are several instances of Markdown linters available with varying feature sets, but I haven't found one that could delegate code blocks to other tools for processing. This isn't really difficult to implement, I have a basic PoC at https://github.com/straight-shoota/md-code-linter (not sure if it works currently, but the code should given an idea how it works).

[j8r]

@straight-shoota It is possible expose all code blocks in a serialized format, like JSON. It won't be hard to do, and external tools can consume it for linting.

I still think to validate at least Crystal code natively would be a good thing. Of course, it is opt-in with a CLI option.

[straight-shoota]

Extracting the code blocks really isn't a problem. A relatively simple grep over the source files should do (alternatively over doc properties in crystal docs --format json). crystal-examples operates like this, too.

I wouldn't integrate this into docs generator. It's a different task and the generator should just work despite any example code issues. It's more closely related to crystal tool format.
Maybe it could be integrated into that or be a separate tool crystal tool check-examples. How exactly can be figured out later. First we need to get the very basic behaviour working. crystal-examples works but as I understand it, requires some amount of manual oversight (see Difficulties). For this features to be any useful, it needs to run in CI.

So we'll have to figure out what's necessary for that. We might have to consider slight changes to the example code format (such annotations to disable checker for code that isn't supposed to compile).
crystal-examples is probably a good basis for that. Let's team up and discuss how to solve the problems it has shown. IMO evolving that tool is better than starting to hack at the compiler. Once the implementation is sufficient, we can consider how to integrate it into the compiler workflow.