Support `-from-tasty` in Dottydoc #4789

Duhemm · 2018-07-13T07:32:09Z

It's not yet integrated with the doc task in sbt, but I think this can be reviewed independently first.

nicolasstucki

Reviewed only compiler and phases. Will do the rest later.

nicolasstucki · 2018-07-18T09:03:53Z

compiler/test/dotty/tools/IOUtils.scala

+    try fn(temp)
+    finally {
+      val allFiles = getAll(temp, "glob:**").sortBy(_.toAbsolutePath.toString).reverse
+      allFiles.foreach(Files.delete(_))


Use dotty.tools.io.Path.deleteRecursively, it does it more efficiently.

nicolasstucki · 2018-07-18T09:06:05Z

compiler/test/dotty/tools/IOUtils.scala

+    val paths = Files.walk(base)
+    val matcher = FileSystems.getDefault.getPathMatcher(pattern)
+    try paths.filter(matcher.matches).iterator().asScala.toList
+    finally paths.close()


dotty.tools.io.Directory.walkFilter does something similar.

We may not need IOUtils at all

nicolasstucki · 2018-07-18T09:14:36Z

doc-tool/src/dotty/tools/dottydoc/DocFrontEnd.scala

+
+  override def runOn(units: List[CompilationUnit])(implicit ctx: Context): List[CompilationUnit] = {
+    if (ctx.settings.fromTasty.value) {
+      val fromTastyFrontend = new ReadTastyTreesFromClasses


A front end should probably not be in another frontend. DocCompiler should choose between DocFrontEnd or ReadTastyTreesFromClasses based on ctx.settings.fromTasty.value

I also wanted to decouple everything, but I ran into issues when cooking the comments. I'll investigate a bit more.

nicolasstucki · 2018-07-18T09:16:40Z

doc-tool/src/dotty/tools/dottydoc/DocFrontEnd.scala

+
+      val typer = new Typer()
+      if (ctx.settings.YcookComments.value) {
+        ctx.docbase.docstrings.keys.foreach { sym =>


Similarly, this should probably be in it's own phase. Which would be added conditionally based on ctx.settings.YcookComments.value.

nicolasstucki · 2018-07-18T09:17:08Z

doc-tool/src/dotty/tools/dottydoc/DocFrontEnd.scala

+      val fromTastyFrontend = new ReadTastyTreesFromClasses
+      val unpickledUnits = fromTastyFrontend.runOn(units)
+
+      val typer = new Typer()


No need to initialize Typer if ctx.settings.YcookComments.value is false

nicolasstucki · 2018-07-18T09:29:39Z

doc-tool/src/dotty/tools/dottydoc/core/DocASTPhase.scala

@@ -27,20 +24,20 @@ class DocASTPhase extends Phase {
  def phaseName = "docASTPhase"

  /** Build documentation hierarchy from existing tree */
-  def collect(tree: Tree)(implicit ctx: Context): Entity = {
+  def collect(tree: Tree)(implicit ctx: Context): List[Entity] = {


Why did you change Entity to List[Entity]?

As far as I can see there are only lists of size 0 or 1.

That's to support multiple @usecase. See the commit messages of c528d6a and f1da1e1

This is where longer lists may be created: https://github.com/lampepfl/dotty/pull/4789/files#diff-5a9874880800484ed4b082c8d776c661R30

nicolasstucki · 2018-08-06T08:17:07Z

test performance please

dottybot · 2018-08-06T08:19:06Z

performance test scheduled: 1 job(s) in queue, 0 running.

The changes have been done

nicolasstucki · 2018-08-06T08:19:46Z

@Duhemm is this still WIP?

dottybot · 2018-08-06T10:16:11Z

Performance test finished successfully:

Visit http://dotty-bench.epfl.ch/4789/ to see the changes.

Benchmarks is based on merging with master (f39943d)

Duhemm · 2018-08-15T08:09:48Z

@nicolasstucki I think it's ready for another pass! Thanks for taking a look.

Dottydoc can now be used in one of two ways: - By receiving a list of source files that will be parsed, typed, and for which the documentation will be generated - When `-from-tasty` is set, by receiving a list of fully qualified class names. The trees will be unpickled and the documentation will be generated.

We're not reusing the PositionPickler for this task, because it is very tied to pickling the positions of `Tree`s, which `Comment`s are not. We simply pickle the coordinates of the comment, and reconstruct the position from the coordinates when unpickling.

The comments are now always pickled as raw comments, as seen in the source file. It's the job of Dotty doc to trigger expansion of the comments after they are unpickled. This approach simplifies incremental generation of the documentation, as we don't need to teach Zinc how to extract dependencies in the documentation.

The `@sourcefile` annotation is added to symbols when they're unpickled from TASTY, and we need to remove them so that they do not appear when generating the documentation.

This special entity was used to mean that an entity needs to be removed from the doc AST after the transformation. Instead, we refactor the DocMiniPhases to have them return a `List[Entity]` (rather than a single `Entity`, as before). It follows that removing an entity from the AST means returning an empty list after transformation. This change will be necessary to be able to support multiple `@usecase` sections in the documentation, because it means that the transformation has to introduce multiple `Entity`-es.

In the documentation, it is possible to have several `@usecase` sections, which means that each of these section should be displayed as a separate member in the documentation. So far, Dottydoc was only considering the first `@usecase` section and was disregarding the others. This commit fixes that, and generates a new member for each of the usecases. Fixes scala#4752

This miniphase is used to cook the comments when `-Ycook-comments` is set. This phase is used by the compiler, the REPL, Dotty IDE and Dottydoc.

Duhemm · 2018-09-04T08:49:22Z

@nicolasstucki Rebased and ready for another round 😄

allanrenucci · 2018-09-04T13:57:26Z

doc-tool/src/dotty/tools/dottydoc/model/factories.scala

@@ -42,7 +36,9 @@ object factories {
  }

  def annotations(sym: Symbol)(implicit ctx: Context): List[String] =
-    sym.annotations.map(_.symbol.showFullName)
+    sym.annotations.collect {
+      case ann if ann.symbol != ctx.definitions.SourceFileAnnot => ann.symbol.showFullName


I don't think this is specific to SourceFileAnnot. We have a bunch of internal annotation that shouldn't be shown in the doc. Maybe !ann.symbol.is(Synthetic) would discard these annotations. Not sure...

allanrenucci · 2018-09-04T14:04:28Z

compiler/src/dotty/tools/repl/ReplCompiler.scala

@@ -196,13 +194,16 @@ class ReplCompiler extends Compiler {
        val stat = stats.last.asInstanceOf[tpd.Tree]
        if (stat.tpe.isError) stat.tpe.show
        else {
-          val docCtx = ctx.docCtx.get


docCtx is now optional in the REPL?

allanrenucci · 2018-09-04T14:06:09Z

compiler/src/dotty/tools/dotc/core/tasty/CommentPickler.scala

@@ -23,7 +23,7 @@ class CommentPickler(pickler: TastyPickler, addrOfTree: tpd.Tree => Option[Addr]
      buf.writeAddr(addr)
      buf.writeNat(length)
      buf.writeBytes(bytes, length)
-      buf.writeByte(if (cmt.isExpanded) 1 else 0)
+      buf.writeLongInt(cmt.pos.coords)


@smarter TASTY version bump?

Duhemm force-pushed the topic/dottydoc-from-tasty branch from 5e37d73 to 8411a66 Compare July 13, 2018 08:40

Duhemm added area:pickling stat:needs review area:doctool area:repl area:ide labels Jul 13, 2018

allanrenucci requested a review from nicolasstucki July 16, 2018 14:36

allanrenucci assigned nicolasstucki Jul 16, 2018

nicolasstucki previously requested changes Jul 18, 2018

View reviewed changes

nicolasstucki reviewed Jul 18, 2018

View reviewed changes

nicolasstucki assigned Duhemm and unassigned nicolasstucki Jul 30, 2018

Duhemm force-pushed the topic/dottydoc-from-tasty branch from 8411a66 to fb098d6 Compare August 2, 2018 14:23

Duhemm added stat:wip and removed stat:needs review labels Aug 2, 2018

allanrenucci added stat:needs rebase and removed stat:wip labels Sep 3, 2018

Duhemm added 7 commits September 4, 2018 09:44

Add tests for Dottydoc from tasty

0c555cc

Remove unused comment property of UseCases

79ffffd

Small refactorings in Comments

a02f8ad

Pickle comments positions

12e4605

We're not reusing the PositionPickler for this task, because it is very tied to pickling the positions of `Tree`s, which `Comment`s are not. We simply pickle the coordinates of the comment, and reconstruct the position from the coordinates when unpickling.

Doc-tool: Remove @sourcefile annotation

8bacff5

The `@sourcefile` annotation is added to symbols when they're unpickled from TASTY, and we need to remove them so that they do not appear when generating the documentation.

Duhemm added 9 commits September 4, 2018 10:05

Cook comments before displaying in the IDE

b2fe7cb

Further refactorings to Comments

1a458ba

More refactorings to comment cooking

66d44a2

Cook documentation when displaying in REPL

c58a4fd

Remove IOUtils

0f5b5ed

Decouple source and TASTY frontend in dottydoc

a082b5a

Add CookComments miniphase

0cd20b4

This miniphase is used to cook the comments when `-Ycook-comments` is set. This phase is used by the compiler, the REPL, Dotty IDE and Dottydoc.

Duhemm force-pushed the topic/dottydoc-from-tasty branch from 6c4ea62 to 0cd20b4 Compare September 4, 2018 08:24

Duhemm assigned nicolasstucki and unassigned Duhemm Sep 4, 2018

nicolasstucki removed the stat:needs rebase label Sep 4, 2018

nicolasstucki approved these changes Sep 4, 2018

View reviewed changes

nicolasstucki assigned Duhemm and unassigned nicolasstucki Sep 4, 2018

allanrenucci reviewed Sep 4, 2018

View reviewed changes

Duhemm merged commit e02c785 into scala:master Sep 4, 2018

allanrenucci mentioned this pull request Sep 5, 2018

Nightly build failure #5079

Closed

Blaisorblade deleted the topic/dottydoc-from-tasty branch September 7, 2018 12:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support `-from-tasty` in Dottydoc #4789

Support `-from-tasty` in Dottydoc #4789

Duhemm commented Jul 13, 2018

nicolasstucki left a comment

nicolasstucki Jul 18, 2018

nicolasstucki Jul 18, 2018

nicolasstucki Jul 18, 2018

nicolasstucki Jul 18, 2018

Duhemm Aug 2, 2018

nicolasstucki Jul 18, 2018

nicolasstucki Jul 18, 2018

nicolasstucki Jul 18, 2018

nicolasstucki Jul 18, 2018

Duhemm Aug 2, 2018

nicolasstucki commented Aug 6, 2018

dottybot commented Aug 6, 2018

nicolasstucki commented Aug 6, 2018

dottybot commented Aug 6, 2018

Duhemm commented Aug 15, 2018

Duhemm commented Sep 4, 2018

allanrenucci Sep 4, 2018

allanrenucci Sep 4, 2018

allanrenucci Sep 4, 2018

Support -from-tasty in Dottydoc #4789

Support -from-tasty in Dottydoc #4789

Conversation

Duhemm commented Jul 13, 2018

nicolasstucki left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicolasstucki commented Aug 6, 2018

dottybot commented Aug 6, 2018

nicolasstucki commented Aug 6, 2018

dottybot commented Aug 6, 2018

Duhemm commented Aug 15, 2018

Duhemm commented Sep 4, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Support `-from-tasty` in Dottydoc #4789

Support `-from-tasty` in Dottydoc #4789