TypeScript is crashed by using too much memory

[falsandtru]

This crash is caused by a few codes. Is this correct behavior?

TypeScript Version:

1.9.0-dev.20160609

Code

https://github.com/falsandtru/spica/tree/v0.0.6
https://travis-ci.org/falsandtru/spica/builds/136372602
https://github.com/falsandtru/spica/blob/v0.0.6/src/lib/monad/sequence.ts

$ $(npm bin)/tsc --noEmit -t es5 --diagnostics typings/tsd.d.ts typings/spica.d.ts typings/lib.ex.d.ts spica.ts

<--- Last few GCs --->

  168134 ms: Mark-sweep 1292.9 (1434.6) -> 1292.9 (1434.6) MB, 3959.0 / 0 ms [allocation failure] [GC in old space requested].
  172283 ms: Mark-sweep 1292.9 (1434.6) -> 1292.9 (1434.6) MB, 4149.0 / 0 ms [allocation failure] [GC in old space requested].
  176418 ms: Mark-sweep 1292.9 (1434.6) -> 1292.9 (1434.6) MB, 4135.0 / 0 ms [last resort gc].
  180433 ms: Mark-sweep 1292.9 (1434.6) -> 1292.9 (1434.6) MB, 4015.0 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 000001F022BC9E79 <JS Object>
    1: objectTypeRelatedTo [*\spica\node_modules\typescript\lib\tsc.js:~17516] [pc=000002F3674B3CA8] (this=000001F022BE4519 <JS Global Object>,source=00000096942236C1 <a Type with map 0000038A0C559FA9>,originalSource=00000096942236C1 <a Type with map 0000038A0C559FA9>,target=000003ED5B64DCC9 <a Type with map 0000031C76BCBA19>,reportErrors=000001F022B04299 <false>)
...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

[mhegazy]

The source of the regression was #8713

[falsandtru]

v1.8.10 has a same problem, and sometimes v1.8.30 with VS2015 will be crashed a few minutes later. So I feel this problem has another reason different from #8713.

[mhegazy]

In 1.8.10, i see it finishing up compilation in less than 10 secs, but uses about 1 GB of memory. this commit is when it started to crash the compiler.

In general, these complex recursive definitions like the ones in https://github.com/falsandtru/spica/blob/v0.0.6/src/lib/monad/sequence.ts, do not fare well in a structural type system. the system needs to relate these generic types structurally, and every time it descends into these types, it instantiates their method signatures, which include other generic types, and so on.

This is similar to the issues we ran into with RxJS a while back, see #7344. the fix was to simplify the complexity of these types to avoid the expensive comparisons, see ReactiveX/rxjs#1475.

[falsandtru]

In short, crashes of VS won't be fixed? VS uses 2.5~3.0GB of memory.

[falsandtru]

I found a solution.

before:

$ $(npm bin)/tsc --noEmit -t es5 --diagnostics typings/tsd.d.ts typings/spica.d
.ts typings/lib.ex.d.ts spica.ts
Files:              34
Lines:           19533
Nodes:           94018
Identifiers:     35814
Symbols:       1964595
Types:          529786
Memory used:   828809K
I/O read:        0.03s
I/O write:       0.00s
Parse time:      0.88s
Bind time:       0.58s
Check time:     28.43s
Emit time:       0.00s
Total time:     29.89s

after:

- export class Sequence<T, S> extends Monad<T> implements ISequence<T, S> {
+ export class Sequence<T, S> extends Monad<T> {

$ $(npm bin)/tsc --noEmit -t es5 --diagnostics typings/tsd.d.ts typings/spica.d
.ts typings/lib.ex.d.ts spica.ts
Files:              34
Lines:           19533
Nodes:           94000
Identifiers:     35807
Symbols:        892012
Types:           41074
Memory used:   119040K
I/O read:        0.03s
I/O write:       0.00s
Parse time:      1.14s
Bind time:       0.62s
Check time:      5.84s
Emit time:       0.00s
Total time:      7.60s

Is this a correct behavior? It seems to leak Symbols and Types. I think, interfaces shouldn't impair the performance.

[mhegazy]

What do you mean by leak symbols?

[falsandtru]

The compiler uses too many symbols, types, memory and Check time only for implementation of interfaces.

- export class Sequence<T, S> extends Monad<T> implements ISequence<T, S> {
+ export class Sequence<T, S> extends Monad<T> {

- Symbols:       1964595
+ Symbols:        892012
- Types:          529786
+ Types:           41074
- Memory used:   828809K
+ Memory used:   119040K
- Check time:     28.43s
+ Check time:      5.84s

[sandersn]

Actually Sequence and ISequence are the same type -- sequence.ts imports Sequence as ISequence from spica, which exports everything from exports.ts, which imports Sequence from sequence.ts. The compiler should issue an error in this case.

Supervisor and ISupervisor have the same problem.

In my testing, spica v0.0.6 still crashes node even with these implements clauses removed.

[falsandtru]

I want to ensure type consistency of declaration(.d.ts) and implementation(.ts) such as class Sequence implements ISequence. How can I ensure this consistency after fixed?

[sandersn]

In the original code, Sequence was not checked against Sequence in spica.d.ts, but against itself. Actually, you don't need to ensure consistency of the implementation with the declaration. Just generate declarations with --declaration when compiling. Then you just need to ensure that the newly generated declaration is the same as the old version. Use source control for that.

For example, compile with --declaration. When you get ready to check in the code, make sure that the generated spica.d.ts has not changed. If it has changed, you'll see a diff and can decide whether the declaration changes are OK to check in.

[falsandtru]

Thanks for the explanation. But I want to manage and design .d.ts file manually because I'm using .d.ts file for API documentation and API visibility control. If you remove the way to ensure type consistency of the implementation with the declaration, it is truly unfortunate.

[sandersn]

I see. You're right, I misread the code -- it's not circular but goes to the d.ts. Unfortunately, typescript definitely does lots of computation to check that two complex structures are structurally identical. I'm not sure there's a way to get around that and still make sure that your implementation perfectly matches your interface.

[falsandtru]

Thanks, I'm waiting for the results of your investigation.

[falsandtru]

@sandersn Any updates? I'm stuck by this bug. Please fix it.

[sandersn]

Unfortunately, we just can't support structural comparison of two gigantic, identical types in the compiler as it is today. The optimisation that I removed in #8713 is incorrect and skips checks that should actually happen. Efficiency would have to come from elsewhere.

The bottom line is that TypeScript can't support your d.ts-checked workflow for this library. As a workaround, you'll have to stop the d.ts-checking or simplify the types somehow.

[falsandtru]

@sandersn This overflow is caused even when I don't use .d.ts files and an implements keyword. Probably it caused by extends keyword. Is this also a design limitation?

[falsandtru]

I'm not clear what is structural comparison of two gigantic identical types? I removed checks using implements, but tsc is still hung.

[falsandtru]

I found another case that is not used structural comparisons.