Add capture checking to some standard library classes #18192

odersky · 2023-07-12T08:33:49Z

Convert some standard library classes to capture checking. The converted classes are represented as tests.
Initially, the Iterator and IterableOnce classes are converted. UPDATE: By now, we have quite a few more, including List, ListBuffer, View, and all their super classes and traits.

Also change capture checker to facilitate the conversion. The main changes are about better interop with classes that are not (yet) capture checked.

Based on #18131

Add unsafeAssumePure method as an escape hatch which is more specific than a cast.

This would become a problem when adding fluid capture sets to FromJavaObject types. And it anyway gives better error messages.

The pickler test did not track whether capture checking was on. This meant that files with import language.experimental.captureChecking has to be compiled additionally with the capture checking setting -language.experimental.captureChecking to pass pickler tests. This made it impossible to test mixed code bases where some files require capture checking and others don't. We now use the previous setting of `CompilationUnit#needsCaptureChecking` when unpickling.

Similar to the treatment of WithPureFuns, we add an annotation WithCaptureChecks to toplevel classes if they were compiled with capture checking on. We also keep a flag on all classes so that we can find out fast whether the class was capture checked or not. However, that flag is not pickled. It is instead reconstituted by the unpickler when it sees the WtihCaptureChecks annotation.

When interacting with files that are not captue checked, we should assume that types can have arbitrary capture sets. This is analogous to interacting with non-null checked files, where we also assume that types in the interface can be null or not. This commit enables interop for normal rechecking by decorating types coming from sources that are not capture-checked with a special `Fluid` capture set. This set both super- and subcaptures any other capture set.

Supress bounds checking in CheckCaptures if the type constructor of an applied type has not been compiled with capture checking.

It seems it's no longer needed.

The original versions of Iterator and IterableOnce before adding capture checking annotations

- Also add Fluid to function types - Add Fluid recursively in arguments of fluidified types - But stop when a capturing type is encountered Also, fix needsVariable for FromJavaObject. FromJavaObject should behave like Any (i.e. no capture set variable needs to be added).

User-defined value classes should be able to capture capabilities

Otherwise we force purity where none might exist.

Members andThen, compose, curried, tupled of function types are now given special types for capture checking that reflect fine-grained capture dependencies.

With the new gradual typing using Fluid capture sets, this should be no longer needed.

compiler/src/dotty/tools/dotc/cc/CheckCaptures.scala

Linyxus · 2023-07-17T09:24:12Z

tests/pos/stdlib/collection/IterableOnce.scala

+  def knownSize: Int = -1
+}
+
+final class IterableOnceExtensionMethods[A](private val it: IterableOnce[A]) extends AnyVal {


Should it be impure? otherwise the extension methods only apply on pure IterableOnces.

I left it like that since all extension methods are deprecated anyway,

As `Map` shows, it's possible that an overriding check is from an external non-capture checked subclass member against a currently compiled & capture-checked superclass member. Hence, we have to add Fluid sets to either part of an overriding check if that part is not compiled with capture checking. This allows to include collection/Map in the set of capture checked stdlib sources.

…ib tests

Plus all supertraits of ListBuffer

There was a subtle difference which causeed the stdlib tests to crash under -Xprint:cc

Without this, we get stray capture sets in match selectors if the match is not exhaustive.

odersky · 2023-07-22T08:59:02Z

Observations of shortcomings so far:

In some instances we will get "cannot establish reference" errors when forming a capture set. For instance in the stringOps test in Test_2.scala.
Sometimes we'd like to have mutable variables capturing cap, but this is forbidden. We could do better by maintaining different universal capabilities at each nesting level, and only forbidding instantiating variables with universal capabilities at deeper level.
We have to assume that arrays are over sealed type variables
We have to ensure transitivity of sealed: A sealed type variable can only be instantiated with other sealed type variables or else with type variables upper bounded by deeply pure types. If there's level checking this has to be integrated.

odersky · 2023-07-22T09:46:19Z

@Linyxus This is about as far as I want to push it for now. Can you give it a rough review? I think you need to look only at the changes to the compiler itself; the test programs don't matter so much. It would be good to get it in fast since there are a lot of accumulated changes and other stuff we do with capture checking should build on these.

Linyxus · 2023-07-22T10:52:03Z

Thanks for the heads-up! I'll do the review right away.

Linyxus

LGTM!

Linyxus · 2023-07-22T17:07:11Z

So it seems that test-windows-full is failing for this PR (which is somehow skipped in the CI of this PR but run after merged into main). The error looks a bit weird: syntax error for files that look like this. These files seem to be aliases to other files, but I'm not sure how they are parsed by the test runner and why they stop working on Windows.

/cc @odersky

odersky · 2023-07-22T17:17:27Z

These are symlinks. Maybe Windows does not know about them and interprets them literally?

odersky · 2023-07-22T17:36:16Z

I pushed a new PR that should fix the problem.

odersky added the cc-experiment Intended to be merged with cc-experiment branch on origin label Jul 12, 2023

odersky requested a review from Linyxus July 13, 2023 07:53

nicolasstucki assigned odersky Jul 13, 2023

odersky force-pushed the add-unsafeAssumePure branch 2 times, most recently from c0a7fc1 to 1613aac Compare July 13, 2023 14:20

odersky changed the title ~~Add unsafeAssumePure method~~ Add capture checking to some standard library classes Jul 13, 2023

odersky force-pushed the add-unsafeAssumePure branch from ddb121e to a32b6d4 Compare July 13, 2023 20:31

odersky mentioned this pull request Jul 13, 2023

A problematic test showing interaction of boxing and inheritance #16415

Closed

odersky force-pushed the add-unsafeAssumePure branch from 1c6c09c to bae1a9a Compare July 14, 2023 10:51

odersky added 21 commits July 15, 2023 09:51

Add unsafeAssumePure method

4beba0a

Add unsafeAssumePure method as an escape hatch which is more specific than a cast.

Avoid spurious dealiasing in alignDependentFunction

18a9d2c

This would become a problem when adding fluid capture sets to FromJavaObject types. And it anyway gives better error messages.

Better interop with non-capture-checked files for override checking

66670c2

Better interop for non-capture-checked file for bounds checking

8f04d3a

Supress bounds checking in CheckCaptures if the type constructor of an applied type has not been compiled with capture checking.

Disable mapJavaArgs when rechecking Apply nodes

80943e4

It seems it's no longer needed.

Capture check first collection classes -- original state

df78771

The original versions of Iterator and IterableOnce before adding capture checking annotations

Capture checked versions of Iterator and IterableOnce

a882fe9

Drop redundant statements

c0bcfbe

Update MimaFilters

714606c

Fix rebase breakage

08deac1

Fix more rebase breakage

9734a36

Test for scala#16415

8140273

Enable test that failed in scala#18168

12735bd

Exclude AnyVal from set of pure base classes

e104f8e

User-defined value classes should be able to capture capabilities

Make assigned types of inlined expressions InferredTypes

1a30250

Otherwise we force purity where none might exist.

Special treatment of types of function members

6c949a9

Members andThen, compose, curried, tupled of function types are now given special types for capture checking that reflect fine-grained capture dependencies.

Don't adapt function and by name types when unpickling

cd1a7b2

With the new gradual typing using Fluid capture sets, this should be no longer needed.

odersky added 2 commits July 16, 2023 18:30

Add View.scala to stdlib test

b23bf45

Add collection/Seq to stdlib tests

fc9bb75

Linyxus reviewed Jul 17, 2023

View reviewed changes

compiler/src/dotty/tools/dotc/cc/CheckCaptures.scala Outdated Show resolved Hide resolved

Linyxus reviewed Jul 17, 2023

View reviewed changes

odersky added 9 commits July 17, 2023 19:25

Add subclasses IndexedSeq, LinearSeq, {mutable,immutable}.Seq to stdl…

fa71a56

…ib tests

Add StrictOptimized ops to stdlib tests

0230d03

Add List and ListBuffer to stdlib test

5543104

Plus all supertraits of ListBuffer

Add StringOps and StringBuilder to stdlib tests

7c642c2

Improve error message for escaping capabilities

74696b9

Align typer and rechecker for Labelled and Bind trees

4b812e3

There was a subtle difference which causeed the stdlib tests to crash under -Xprint:cc

Use InferredTypeTree for @unchecked match selectors

0a8f763

Without this, we get stray capture sets in match selectors if the match is not exhaustive.

Add test programs to stdlib tests

bd8a4de

Rename WithCaptureChecks to CaptureChecked

8ce0521

odersky assigned Linyxus and unassigned odersky and Linyxus Jul 22, 2023

Update MiMaFilters

7a8fae7

Linyxus approved these changes Jul 22, 2023

View reviewed changes

Linyxus merged commit 1254e14 into scala:main Jul 22, 2023

Linyxus deleted the add-unsafeAssumePure branch July 22, 2023 15:10

Linyxus restored the add-unsafeAssumePure branch July 22, 2023 17:30

Kordyjan added this to the 3.4.0 milestone Aug 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add capture checking to some standard library classes #18192

Add capture checking to some standard library classes #18192

odersky commented Jul 12, 2023 •

edited

Loading

Linyxus Jul 17, 2023

odersky Jul 19, 2023

odersky commented Jul 22, 2023 •

edited

Loading

odersky commented Jul 22, 2023

Linyxus commented Jul 22, 2023 •

edited

Loading

Linyxus left a comment

Linyxus commented Jul 22, 2023 •

edited

Loading

odersky commented Jul 22, 2023

odersky commented Jul 22, 2023

Add capture checking to some standard library classes #18192

Add capture checking to some standard library classes #18192

Conversation

odersky commented Jul 12, 2023 • edited Loading

Linyxus Jul 17, 2023

Choose a reason for hiding this comment

odersky Jul 19, 2023

Choose a reason for hiding this comment

odersky commented Jul 22, 2023 • edited Loading

odersky commented Jul 22, 2023

Linyxus commented Jul 22, 2023 • edited Loading

Linyxus left a comment

Choose a reason for hiding this comment

Linyxus commented Jul 22, 2023 • edited Loading

odersky commented Jul 22, 2023

odersky commented Jul 22, 2023

odersky commented Jul 12, 2023 •

edited

Loading

odersky commented Jul 22, 2023 •

edited

Loading

Linyxus commented Jul 22, 2023 •

edited

Loading

Linyxus commented Jul 22, 2023 •

edited

Loading