-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix #6152: Dotty tries to override Java bridge methods #6266
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hello, and thank you for opening this PR! 🎉
All contributors have signed the CLA, thank you! ❤️
Have an awesome day! ☀️
This works but I wonder if instead we could change ClassfileParser to not create symbols for bridge methods. This way joint and separate compilation would see the same symbols and this reduces the chance that they'll behave differently. |
We should find out what Scala-2 is doing. It might be that we ignore bridge methods before erasure but make them visible afterwards. |
@odersky So Scala 2 has a very peculiar handling of bridges, as I noted in #6152 (comment):
All in all, it seems simpler to try to align what we do for separate and joint compilation by just not creating bridge symbols in the first place. |
There may be two separate considerations here: the consistency of the Dotty separate compilation with Scala 2 and the consistency of the Dotty separate compilation with the Dotty joint compilation. IMO the internal consistency of the project should be of a higher priority than the external one. The former one yields bugs if not observed, the latter one yields compatibility issues. We could FTTB make the separate compilation consistent with the joint compilation to get rid of the bug while creating a task (an issue, a milestone etc) to get consistent with Scala 2. |
@@ -96,7 +96,8 @@ class CompilationTests extends ParallelTesting { | |||
"tests/neg-custom-args/fatal-warnings/xfatalWarnings.scala", | |||
defaultOptions.and("-nowarn", "-Xfatal-warnings") | |||
) + | |||
compileFile("tests/pos-special/typeclass-scaling.scala", defaultOptions.and("-Xmax-inlines", "40")) | |||
compileFile("tests/pos-special/typeclass-scaling.scala", defaultOptions.and("-Xmax-inlines", "40")) + | |||
compileDir("tests/pos-special/i6152", defaultOptions, separateCompilation = true) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why use pos-special ? If you put i6152 in tests/pos it should work (except that decompilation tests are broken when java files are involved, we work around this by putting the tests in tests/pos-java-interop)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Indeed it worked. It seems compileFilesInDir
does not treat the sources in its base directory as the part of the same compilation, which confused me. Makes sense.
It looks like we can't trivially make Dotty bridge-blind because Dotty makes use of the bridge information. Consider the following line: val f: (=> Int) => Unit = a => () Normally it should compile to the following bytecode: public void main(java.lang.String[]);
Code:
0: aload_0
1: invokedynamic #47, 0 // InvokeDynamic #0:applyVoid:(LTest$;)Ldotty/runtime/function/JProcedure1;
6: astore_2
7: return With the bridge-blindness (the modified fix proposed in this PR), it compiles to: public void main(java.lang.String[]);
Code:
0: aload_0
1: invokedynamic #47, 0 // InvokeDynamic #0:apply:(LTest$;)Ldotty/runtime/function/JProcedure1;
6: astore_2
7: return In the former case, During the bytecode generation, Now, ConsiderationsSo far the issue above is specific to the However, the example reveals that the bridges may potentially provide the information on whether a method is abstract or concrete. If we make Dotty bridge-blind to make the separate compilation similar to the joint compilation, we'll have to explicitly perform the mentioned checks whenever we need them. This may impact performance and give us more headache when we forget to do so. So my opinion is that it'd be best to make all of the available information exposed to Dotty. |
I think we should do exactly what scalac does. It's a tricky subject and scalac had 10 years to refine it. |
I'm not convinced, based on what you're saying it seems that we've always had a bug under joint compilation and that making separate compilation behave like joint compilation is just now exposing this bug. Consider that when dotty-library is compiled, |
Also consider that if |
I think there's just no way we can accurately compute which members are abstract after erasure, because inevitably there's bridges we cannot see (under separate compilation, we cannot see bridges from other Scala files (because pickling is before bridge generation), and under joint compilation, we cannot see bridges from other Java files (because we drop Java compilation units after Typer)). So |
And longer term we should consider changing the definition of the |
Joint compilation works fine since we can determine whether a method is abstract by analysing Java files. We do not need the bridges that come from the class files there.
Can you please elaborate? I thought the order of the phases is irrelevant when using separate compilation since the second compilation run happens after the first one has completed. Hence during the second run, we'll have the class files with the bridges in them.
Ok, it seems we need to solve the #6152 in the context/after #6136. So I'll close this PR FTTB. |
To determine whether a method in a base class is abstract or not, you need to compute overriding relationship, and what I'm saying is that this cannot be done correctly after erasure.
For Dotty classes, we do not unpickle symbols based on the information in the classfile, we only look at the information stored in tasty files, and for Scala 2 classes we only look at the information in the ScalaSig attribute. So in both of these cases we're not going to see any bridge.
I don't think any of the improvement in the Scala 2 ClassfileParser are going to change anything to the situation here. The real solution is in #6266 (comment). |
I believe when we are doing the joint compilation, we are working before the erasure?
So we are still getting the new information different from the one available for the Java-derived class files.
Could be, we never know until we try with such things :) I guess we still need to bring ourselves up to date with Scala 2 since we do not want a divergence here. So if this one can wait, I think it'd be best to avoid tactical fixes here. Even if Scala 2 improvements don't fix this one, we still need to synergise whatever we come up with whatever Scala 2 did. |
|
Yes, but it is working on the symbols generated before the erasure. |
And some of these symbols are going to come from Scala 2 or Dotty classfiles, so we won't generate bridges for them, so the overriding relationship after erasure may incorrectly show some methods as being abstract. |
The point is not the bridges and not the overriding relationship. The point is the symbols being identified as deferred. This is because of In different scenarios, we have different info in our disposal: bridges, tasty, Java sources, etc. The idea is to use this info to prove that a method is deferred. Some of this info is enough to mark a method as deferred, some may be not. I guess inconsistency or impossibility to derive "deferred" from given info is a tricky subject. |
|
True, I believe it is quite likely that to mark a symbol as deferred, overriding needs to be taken into account. I also believe that it is likely that the info supplied must be sufficient to correctly implement inheritance both at compile time and at runtime: inheritance is fundamental to Java, and Java's been around for a long time, so it makes sense for them to get it right. I also believe that if for some reason it is not possible to do so in Scala, we should consider that Scala itself may be buggy. However, I am not familiar with the specifics, so I won't speculate further. Whether deferrence indeed involves an explicit computation of overriding, and whether it is indeed fundamentally impossible to derive it from the info we have is an interesting topic. So if you strongly believe it is the case, maybe you can create an issue so that it is on file and we can work with this later? |
9dbade3
to
62d5f3a
Compare
@@ -375,6 +375,6 @@ object ClassfileConstants { | |||
override def baseFlags(jflags: Int) = if ((jflags & JAVA_ACC_FINAL) == 0) Mutable else EmptyFlags | |||
} | |||
val methodTranslation: FlagTranslation = new FlagTranslation { | |||
override def baseFlags(jflags: Int) = if ((jflags & JAVA_ACC_BRIDGE) != 0) Bridge else EmptyFlags | |||
override def baseFlags(jflags: Int) = if ((jflags & JAVA_ACC_BRIDGE) != 0) BridgeArtifact else EmptyFlags |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This shouldn't be needed since all bridges are emitted with the ACC_SYNTHETIC flag which corresponds to Artifact in Scala. The problem seems to be that we don't actually set the Artifact flag for them: https://github.com/lampepfl/dotty/blob/9dbade3e3c08a81a2b2744d3afa5a8434bd0cdba/compiler/src/dotty/tools/dotc/core/classfile/ClassfileConstants.scala#L344 compare with https://github.com/scala/scala/blob/a26aeb7472ae2d2bab331bfcf9f785c3c7d1c18a/src/reflect/scala/reflect/internal/ClassfileConstants.scala#L350
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do they set it explicitly then at https://github.com/scala/scala/blob/a26aeb7472ae2d2bab331bfcf9f785c3c7d1c18a/src/reflect/scala/reflect/internal/ClassfileConstants.scala#L382?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No particular reason I think, since no one ever generates non-synthetic bridges it doesn't make a difference (but non-bridge synthetic do exist so fixing that is important)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, seems to be working without that line
@@ -1141,7 +1141,7 @@ object SymDenotations { | |||
* either as overrider or overridee. | |||
*/ | |||
final def memberCanMatchInheritedSymbols(implicit ctx: Context): Boolean = | |||
!isConstructor && !is(Private) | |||
!isConstructor && !is(Private) && !is(Artifact) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
!isConstructor && !is(Private) && !is(Artifact) | |
!isConstructor && !is(Private | Artifact) |
@@ -40,7 +40,7 @@ object OverridingPairs { | |||
* relative to <base>.this do | |||
*/ | |||
protected def matches(sym1: Symbol, sym2: Symbol): Boolean = | |||
sym1.isType || self.memberInfo(sym1).matches(self.memberInfo(sym2)) | |||
sym1.isType || (!exclude(sym2) && self.memberInfo(sym1).matches(self.memberInfo(sym2))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this change needed ? Doesn't the existing logic already take care of calling exclude when necessary ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're right, seems to be working without this change
d19d555
to
9fdeb94
Compare
@@ -20,14 +20,15 @@ object OverridingPairs { | |||
/** The cursor class | |||
* @param base the base class that contains the overriding pairs | |||
*/ | |||
class Cursor(base: Symbol)(implicit ctx: Context) { | |||
class Cursor(base: Symbol, excludeArtifacts: Boolean = true)(implicit ctx: Context) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems a bit roundabout to me. Instead of creating symbols for artifacts just to ignore them afterwards, I think we should just not create symbols for artifacts in ClassfileParser. Given that we don't see these symbols under joint compilation it should reduce the differences in execution between joint and separate compilation and thus reduce the number of unique bugs to one or the other.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I propose to make this into a separate issue since it seems to be a tricky topic.
Dotty overrides Java bridge methods when the user intends to override the method for which the bridge was generated. This commit forbids Dotty to override bridge methods altogether.
What happened is:
That test: object Test {
def main(args: Array[String]): Unit = {
val sb = new java.lang.StringBuilder()
sb.append(Array[Char](), 0, 0)
sb.length
}
} The master version translates the invokevirtual #74 // Method java/lang/StringBuilder.length:()I This PR version translates it to: invokevirtual #76 // Method java/lang/AbstractStringBuilder.length:()I At a glance, looks like the master version uses a bridge and this PR version goes for the second-best alternative, the bridge being absent... |
I traced ClassfileParser with: def run()(implicit ctx: Context): Option[Embedded] = try {
ctx.debuglog("[class] >> " + classRoot.fullName)
parseHeader()
this.pool = new ConstantPool
val res = parseClass()
if (Set("java.lang.AbstractStringBuilder", "java.lang.StringBuilder").contains(classRoot.fullName.toString)) {
val length = classRoot.info.decls.filter(_.name.toString.contains("length")).head
println(s"""
|Parsed ${classRoot.fullName}
|Length: ${length}
|Type: ${length.info.show}
|Bridge: ${length.is(Flags.Bridge)}
""".stripMargin)
}
[...] Output on the above test compilation:
So it seems the |
Interesting, javac doesn't do this, so we're not imitating it well enough here: class i6266 {
public void foo(StringBuilder sb) {
System.out.println(sb.length());
}
} public void foo(java.lang.StringBuilder);
descriptor: (Ljava/lang/StringBuilder;)V
flags: ACC_PUBLIC
Code:
stack=2, locals=2, args_size=2
0: getstatic #2 // Field java/lang/System.out:Ljava/io/PrintStream;
3: aload_1
4: invokevirtual #3 // Method java/lang/StringBuilder.length:()I
7: invokevirtual #4 // Method java/io/PrintStream.println:(I)V |
javac sometimes generates bridges to make a method visible (and thus prevent IllegalAccessError), see http://stas-blogspot.blogspot.com/2010/03/java-bridge-methods-explained.html for some examples. |
So it seems here the bridge is generated because |
I propose that in the |
Not quite. The theory says that ignoring synthetic methods coming from classfiles should make separate compilation more like joint compilation, and indeed we can reproduce the bug on master with joint compilation :). package pkg;
class P {
public void foo() {
}
}
public class A extends P {
} object B {
def main(args: Array[String]): Unit = {
val a = new pkg.A
a.foo()
}
} % dotc A.java B.scala
% javac A.java
% dotr B
Exception in thread "main" java.lang.IllegalAccessError: tried to access class pkg.P from class B$
at B$.main(B.scala:4)
at B.main(B.scala) And indeed we use the wrong signature in the method call: 9: invokevirtual #35 // Method pkg/P.foo:()V Compare with scalac: 9: invokevirtual #21 // Method pkg/A.foo:()V |
Ok, you're right, this does look like an issue. Probably we can try this solution again after solving that issue. |
On the second thought, even with the issue fixed, I think we'll still have to retain the bridges when parsing class files. The method calls in question target exactly the bridges, and hence the compiler must be aware of them. Either predict their existence in the joint compilation scenario, or obtain them from the class file on separate compilation. The point is, we do use the bridges on compilation, hence we can't drop them. The IllegalAccessError on joint compilation arises precisely from the fact that the compiler is not aware of the bridges on joint compilation. |
If we can predict them, then we don't need to obtain them in any other way. (and the prediction part is probably not very complicated in practice, we just need to figure out what scalac is doing exactly). |
I would say the reverse is more efficient. In case of joint compilation we have no other choice but predicting them. In case of classfiles, they are already there, so why put extra effort in predicting them? |
Because I suspect that the "prediction" will actually be really fast. Because in the backend you don't know which symbols came from Java sources and which came from Java classfiles so you can't treat them differently, and most important of all because using different ways to achieve the same result increases the chance of bugs |
This is a port of scala/scala@765eb29: When emitting a virtual call, the receiver in the bytecode cannot just be the method's owner (the class in which it is declared), because that class may not be accessible at the callsite. Instead we use the type of the receiver. This was basically done to fix - aladdin bug 455 (9954eaf) - SI-1430 (0bea2ab) - basically the same bug, slightly different - SI-4283 (8707c9e) - the same for field reads In Dotty, we rarely encountered this issue because under separate compilation, we see the bridge symbols generated by javac and use their owner as the receiver, but under joint compilation of .java and .scala sources this doesn't happen, in particular this commit fixes scala#6546. It also means that we should now be able to stop creating symbols in the ClassfileParser for Java bridges, this would make joint and separate compilation more similar and thus should reduce the number of bugs which appear in one but not the other as discussed in scala#6266. After this commit, some more changes are necessary to get the updated backend to work correctly with dotty, these are implemented in the later commits of this PR. Co-Authored-By: Lukas Rytz <[email protected]>
This is a port of scala/scala@765eb29: When emitting a virtual call, the receiver in the bytecode cannot just be the method's owner (the class in which it is declared), because that class may not be accessible at the callsite. Instead we use the type of the receiver. This was basically done to fix - aladdin bug 455 (9954eaf) - SI-1430 (0bea2ab) - basically the same bug, slightly different - SI-4283 (8707c9e) - the same for field reads In Dotty, we rarely encountered this issue because under separate compilation, we see the bridge symbols generated by javac and use their owner as the receiver, but under joint compilation of .java and .scala sources this doesn't happen, in particular this commit fixes scala#6546. It also means that we should now be able to stop creating symbols in the ClassfileParser for Java bridges, this would make joint and separate compilation more similar and thus should reduce the number of bugs which appear in one but not the other as discussed in scala#6266. After this commit, some more changes are necessary to get the updated backend to work correctly with dotty, these are implemented in the later commits of this PR. Co-Authored-By: Lukas Rytz <[email protected]>
This is a port of scala/scala@765eb29: When emitting a virtual call, the receiver in the bytecode cannot just be the method's owner (the class in which it is declared), because that class may not be accessible at the callsite. Instead we use the type of the receiver. This was basically done to fix - aladdin bug 455 (9954eaf) - SI-1430 (0bea2ab) - basically the same bug, slightly different - SI-4283 (8707c9e) - the same for field reads In Dotty, we rarely encountered this issue because under separate compilation, we see the bridge symbols generated by javac and use their owner as the receiver, but under joint compilation of .java and .scala sources this doesn't happen, in particular this commit fixes scala#6546. It also means that we should now be able to stop creating symbols in the ClassfileParser for Java bridges, this would make joint and separate compilation more similar and thus should reduce the number of bugs which appear in one but not the other as discussed in scala#6266. After this commit, some more changes are necessary to get the updated backend to work correctly with dotty, these are implemented in the later commits of this PR. Co-Authored-By: Lukas Rytz <[email protected]>
Dotty overrides Java bridge methods when the user intends to override
the methods for which the bridges were generated. This commit forbids
Dotty to override bridge methods altogether.
This fix rests on the assumption that it is never legal to override synthetically generated bridge methods from the user's code.
Also, it seems that the current test framework does not allow to create tests for this issue. I haven't found a way to reproduce the issue as specified here with the existing test framework.