Hangs forever on simple YAML file when re-building SyntaxSet

[sharkdp]

How to reproduce?

From the syntect root folder, perform these steps:

1. Create a simple YAML file:

   echo "# test" > test.yml

2. Run syncat on this file. Everything works as expected:

   ▶ cargo run --example=syncat -- test.yml
       Finished dev [unoptimized + debuginfo] target(s) in 0.08s
        Running `target/debug/examples/syncat test.yml`
   # test
   

3. Create an empty folder

   mkdir empty
   

4. Run syncat again with --extra-sntaxes set to the empty folder

   ▶ cargo run --example=syncat -- test.yml --extra-syntaxes empty
       Finished dev [unoptimized + debuginfo] target(s) in 0.08s
        Running `target/debug/examples/syncat test.yml --extra-syntaxes empty`
   

   This seems to hang forever.

Additional remarks

• This bug only appears if the YAML file contains a line that begins with a comment
• I could not reproduce this behavior with another syntax.

[sharkdp]

Apparently, it loops endlessly here:

syntect/src/parsing/parser.rs

Lines 200 to 208 in 8238d80

	while self.parse_next_token(
	line,
	syntax_set,
	&mut match_start,
	&mut search_cache,
	&mut regions,
	&mut non_consuming_push_at,
	&mut res
	) {}

with this state:

line = "# test\n"
match_start = 0
regions = Region { raw: OnigRegion { allocated: 10, num_regs: 1, beg: 0x7f6bc34f0a90, end: 0x7f6bc34f0ac0, history_root: 0x0 } }
non_consuming_push_at = (0, 14532)
res = [(0, Push(<source.yaml>))]

The second number in non_consuming_push_at keeps getting incremented. Everything else seems to stay the same.

[keith-hall]

Commenting out this line:

syntect/src/parsing/syntax_set.rs

Line 436 in 8238d80

context.prototype = Some(prototype_id.clone());

seems to fix it, so maybe it's a problem with how prototypes are handled on syntax sets that get turned into builders and back again? Interestingly, the YAML syntax definition's comment context is the last one that gets loaded (due to it being the last context in the YAML.sublime-syntax file), in case that makes any difference.

[keith-hall]

Actually, more precisely, the problem seems to be that the YAML syntax includes the comment context in it's prototype, and it is not being correctly ignored / marked as "no prototype" when rebuilding the syntax set, which is why the stack size is growing constantly.

[keith-hall]

I tried adding:

Pattern::Include(ContextReference::Direct(ref s)) => {
    Self::recursively_mark_no_prototype(syntax, s.index(), contexts, no_prototype);
}

at

syntect/src/parsing/syntax_set.rs

Line 492 in 8238d80

_ => (),

, but it still hangs forever - maybe the indexes are different when rebuilding so the direct references are no longer valid?

[robinst]

Thanks for investigating! Sounds like my bad, I'll have a look tomorrow.

[robinst]

Raised a PR with a fix: #221

[robinst]

@keith-hall: I think all your fix was missing was doing the same thing for references in Pattern::Match :).

[keith-hall]

heh I was so close, thanks for taking over the investigation and fixing the problem @robinst - your test coverage is awesome, I could think of no suggestions for improvement ;)