Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "Typings" file to NPM package for Typescript consumers #5717

Closed
thw0rted opened this issue Aug 2, 2017 · 39 comments · Fixed by #8878
Closed

Add "Typings" file to NPM package for Typescript consumers #5717

thw0rted opened this issue Aug 2, 2017 · 39 comments · Fixed by #8878

Comments

@thw0rted
Copy link
Contributor

thw0rted commented Aug 2, 2017

I've been working on a Cesium-based project for a few months now, and picked Visual Studio Code as my primary editor. I was a little irritated that other libraries I rely on (like Moment and jQuery) have good feedback from Intellisense, while Cesium doesn't give me anything (out of the box, at least). I learned that this is because the more popular libraries have a "Typings" file (.d.ts) either available from an external source, or bundled into the Node module directly.

I found this repo and placed the index.d.ts in my node_modules/cesium/ and like magic, VSCode knew the types of all the Cesium classes and functions -- except that listing was last updated 4 months ago with version 1.26, so it was missing stuff that's changed in the last half-dozen or so releases. I've added some things to it myself, though I wouldn't say it's definitely complete. I can post my modified version if anybody would like that.

I recognize that this would be an ongoing requirement, either to maintain manually or to generate as part of your build workflow, but I think it would really help out both those that try to use Cesium from actual Typescript, and those that work with vanilla-JS Cesium from VSCode.

@ggetz
Copy link
Contributor

ggetz commented Aug 2, 2017

Hi, thanks @thw0rted! There's been previous discussion about starting to use Typescript (see #4434) and it's included on our BIM Roadmap if you'd like to contribute to discussions there.

@hpinkos
Copy link
Contributor

hpinkos commented Aug 2, 2017

We've been talking about updating to es6 modules sometime soon, so we may be able to do this as part of that effort.

@mramato
Copy link
Contributor

mramato commented Aug 2, 2017

ES6 is orthogonal to this.

The best way to implement this would be to automatically generate it from the jsDoc comments, which is what some of our users have tried doing. (I believe jsDoc already outputs an interim json document that makes this straightforward). Hopefully there is a standard tool available that we can use to validate the output so we can automatically detect when it's broken as well. I imagine there are bugs in the jsDoc that would have to be cleaned up in the process as well.

It's doubtful that the core team will be able to do this anytime soon, but if someone would like to implement a gulp task that generates the definition file, we would be happy to take the contribution, it just needs to be auto-generated and cleanly written so that it's minimal work for us to maintain going forward.

@thw0rted
Copy link
Contributor Author

thw0rted commented Aug 2, 2017

I read #4434, and while moving to TS would be neat and all, I'm talking about "just" adding typings sooner rather than later. As Matt points out I think this is reasonably straightforward if the JSDoc is already correct (and of course, if the JSDoc isn't correct, it should be updated).

In the past few days working with it, the biggest hangup has been dealing with Property types. I don't know a lot of Typescript so if there's a way to say that you can assign any other type (or specific other types, declared as a generic like Property<boolean>) and implicitly convert, I haven't found it. I updated a lot of type members that are listed as just Property in the docs to be Property | Color (or boolean or number or whatever), just so that assignments work. I'll look into the toolchain to generate the typings file from a gulp task, but we need to figure out the Property thing before the output will really be useful.

@thw0rted
Copy link
Contributor Author

thw0rted commented Aug 2, 2017

Oh! And I forgot, while ES6 is kinda-sorta orthogonal, I believe that importing an ES6 module in VSCode will apply Typescript-style static checking to the JSDoc (if intact) which would mean you wouldn't strictly need a typings file to get Intellisense working. (You might still need the file to actually include Cesium in a Typescript project without copping out and calling it any?)

@thw0rted
Copy link
Contributor Author

thw0rted commented Aug 2, 2017

I'm trying to rectify how Cesium handles Property with how Typescript wants this to work.

For example, in a simple Billboard Sandcastle, I can set viewer.entities.billboard.show = false and it appears to be setting a property of the billboard. Under the hood, it looks like the setter is creating a ConstantProperty and saving that to a private member. This is a common pattern across the sample code.

Typescript has some strong opinions about this. From that issue:

I truly object to the idea that this code foo.bar = "hello"; console.log(foo.bar); should ever print anything than "hello" in a language that attempts to have sane semantics.
Properties should have behavior that is indistinguishable from fields to external observers.

This is on an issue talking about why Typescript won't let type types of a getter/setter pair differ from each other. So there's currently no way to declare typings such that billboard.show = true; works and so does if(defined(billboard.show) && billboard.show.getValue()){ ... }. You can declare show as being of type Property | boolean (which is exactly what I did for the constructor options) but that means that you have to add a type guard any time you get the property, even though you know it will always be a Property and not a raw boolean.

@mramato
Copy link
Contributor

mramato commented Aug 2, 2017

While I don't have a good answer for you, what Cesium is inspired by another Microsoft language, C# and is normally referred to as implicit construction. You can read more about it here: https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/implicit It's not that the setter/getter are different types, it's that the Property type is automatically instantiated during the assignment. Conceptually, you are always setting a Property type, it's just created on the fly.

I don't know enough about TypeScript to know if it has any sort of implicit construction, but if there is any such concept, I think that could be used here. Ideally, in the TS definition, it should always say Property{bool} (or whatever the generic notation is) is the type for the getter/setter, but it would know that Property can be implicitly cast from bool.

Of course this is me talking in the abstract and unlikely to have a good TS solution, but I just wanted to provide some context as to the idea behind the properties.

I also wonder if we can lie to TS for the sake of a good definition file, even if the actual implementation is slightly different.

@thw0rted
Copy link
Contributor Author

thw0rted commented Aug 2, 2017

I did a little research about implicit cast / implicit construction but didn't have time to really get into it. I'm familiar with the idea, and the good news is that Anders Hejlsberg worked on both languages so there's probably support for getting the feature in there if it's not already. I believe some primitive types can already be implicitly assigned / cast to one another, so hopefully there's some way to shoehorn in implicit construction too.

Maybe I should take this to the TypeScript tracker for more details...

@thw0rted
Copy link
Contributor Author

If anybody is tracking, I went ahead and hand-jammed some 1.37.0 updates to the Typings file I found earlier: https://gist.github.com/thw0rted/c0c73335c9968d5e7b6dd18a43ef6c42

I'm sure it's still missing some information or wrong in places, and I don't have automated tooling in place to generate this, which is what's really needed. Still no workaround as far as I know for the issue I raised back in August -- Typescript still really doesn't want you to use getter/setter pairs the way Cesium does, so you might have to do use type guards or casts in your consuming code that wouldn't otherwise be necessary.

@emackey
Copy link
Contributor

emackey commented Oct 10, 2017

I stumbled across another thing to consider here: TypeScript without Transpiling.

TypeScript 2.3, which arrived in April, introduced support for analyzing conventional JavaScript code that has type annotations in comments. You can use a JSDoc-inspired syntax to describe function signatures and add type information. TypeScript tools read the type annotations from the comments and use them in much the same way that they use TypeScript’s own type system.

JavaScript with type annotations in comments is less succinct than writing actual TypeScript code, but it works everywhere, it eliminates the need for transpiling, and it makes the TypeScript tooling totally optional. It doesn’t cover all the same ground as the TypeScript language yet, but it is comprehensive enough to be useful.

@beginor
Copy link

beginor commented Oct 13, 2017

@thw0rted I have move your gist to a public repo https://github.com/beginor/cesium-typings , this make it easy for others to contribute.

Any star or fork is welcome.

@pjcozzi
Copy link
Contributor

pjcozzi commented Jan 6, 2018

Also see https://github.com/excelulous/cesium-type-definitions by via @excelulous https://groups.google.com/forum/#!topic/cesium-dev/25jPqr5ODg0

If the community wants to collaborate on a single TypeScript definition for Cesium - and even contribute it back in a way that is easy for the core team to maintain - please go for it!

@bampakoa
Copy link
Contributor

Hi all,

I am currently experimenting with tsd-jsdoc to produce a typings file for CesiumJS. I have finally managed to create an output file (not sure if it is ok, needs review) but I needed to make a bunch of modifications so that the tool can be aligned with CesiumJS. In detail:

  • When I run the tool, I noticed some inconsistencies between the primitive types that are defined in Cesium and those that the library expects to parse. For example: the tool looks for string, number, boolean and not String, Number, Boolean.
  • In some cases the error handling was not so good and the library was crashing if it could not find some properties. I added some code there also to handle exceptions.

The library does not seem to be stable yet. It had stuck for a year on 2.0.0-beta.1 version and suddenly there were some new pre-releases out in the last 15 days. I do not know if it is worth to make PRs to fix their current code.

@pjcozzi what do you think about maintaining a tailored made fork of this tool, for the needs of CesiumJS? Alternatevily, I could try to make CesiumJS be aligned with their current codebase by submitting PRs for the current Cesium documentation.

I really want to help on this so in case someone is interested, we can start collaboration and finally take something under review from the Cesium team.

A thread was also started on the Cesium forum if you want to have a look. https://groups.google.com/forum/#!searchin/cesium-dev/Angular%7Csort:date/cesium-dev/Ztd5jNcKPdo/z1AHGEfKAAAJ

@szechyjs
Copy link

I recently published types with DefinitelyTyped. They can be found in the @types/cesium package.

https://www.npmjs.com/package/@types/cesium

@thw0rted
Copy link
Contributor Author

@szechyjs How did you generate these typings? I saw in the NPM listing that you credit @Zuzon so I assume you're starting from the same hand-massaged version I've been maintaining in a gist. I try to keep up with changes that are mentioned in the changelog but I'm sure there's plenty of stuff I've missed. It's good to have something in DT but I think in the long run @bampakoa 's approach (generating typings from existing JSDoc) is really the only way the typings are going to stay accurate, given the significant rate of change this project sees.

@mramato
Copy link
Contributor

mramato commented Jun 13, 2018

@bampakoa First off, thanks for your interest in Cesium and desire to help the community!

what do you think about maintaining a tailored made fork of this tool, for the needs of CesiumJS?

Our usual preference is to submit patches to other tools upstream, that way we help improve them for everyone and play the role of good OSS citizen. If customization is needed for Cesium, then having plug-in points for the tsd-jsdoc would be preferred over a fork. That being said, I haven't looked at tsd-jsdoc, if it's in the realm of a single file, then just having a generate-tsd gulp task is a possibility, in which case we can just have the whole code base be in Cesium.

Alternatevily, I could try to make CesiumJS be aligned with their current codebase by submitting PRs for the current Cesium documentation.

For example: the tool looks for string, number, boolean and not String, Number, Boolean.

Since JSDoc is perfectly fine with String/Number/Boolean, I assume that any tool that claims jsdoc -> tsd should also be okay with it. So this is something I would fix in the tool, not in Cesium. If you find invalid JSDoc in Cesium, then Cesium should definitely be fixed, but other than that, we would want to keep changes to Cesium to a minimal.

Ultimately, the Cesium team's preference would be to generate official bindings automatically as part of our release process (and travis builds). I know there have been separate efforts from those in the community to generate tsds, but I don't think anyone has ever opened a Pull Request to make it a reality.

Given Cesium's scope, there's no way a PR is going to get merged without a lot of discussion and iteration. @bampakoa since this seems like something that really interested you, I would suggest opening a PR, even if it's only 50% done in order to get feedback from the maintainers and community regarding the approach you're taking and quality of the output. Sounds like @thw0rted and @szechyjs both may have some good insights and ideas here (as well as previous hurdles they have had to overcome in their own efforts). Perhaps you can even rally others to open PRs to your PR to help further improve it. I'm sure it will expose issues in Cesium's JSDoc code as well.

I personally would love to do more with TypeScript, but none of the libraries I work on currently use it (This is true for most (all?) of the Cesium maintainers.) and it's unlikely Cesium itself will ever switch to it. However, we want to support our community of TS Cesium devs so I would love to see this happen.

@thw0rted
Copy link
Contributor Author

thw0rted commented Jun 13, 2018

For the question about string/number/boolean, this is a peculiarity of TypeScript -- see this SO answer which references this documentation:

Don’t ever use the types Number, String, Boolean, or Object. These types refer to non-primitive boxed objects that are almost never used appropriately in JavaScript code.

That is, number is the type of the primitive literal 42 and Number is the class that holds e.g. the toFixed function. If tsd-jsdoc is unhappy about function arguments that are defined as boxed-type, it might just be codifying this guidance from the TS handbook as a "rule".

ETA: I was looking for this counterexample or something like it -- it shows how you can't assign an object-type to a primitive-type. I'm still not 100% clear on why, but I do stick to the advice about using primitives over objects where plausible.

@mramato
Copy link
Contributor

mramato commented Jun 13, 2018

That is, number is the type of the primitive literal 42 and Number is the class that holds e.g. the toFixed function. If tsd-jsdoc is unhappy about function arguments that are defined as boxed-type, it might just be codifying this guidance from the TS handbook as a "rule".

Right, what I'm suggesting is that tsd-jsdoc should treat Number in JSDoc as number in TypeScript.

@thw0rted
Copy link
Contributor Author

I don't know enough TS (or JS for that matter) to tell you if that's safe. The primitive/object-box distinction is not unique to TS, it exists in JS as well -- see e.g. the MDN docs. Cesium functions almost definitely don't require a boxed object, so I think they should probably specify the unboxed primitive, but I'm not personally expert enough in why to argue about it here.

@mramato
Copy link
Contributor

mramato commented Jun 13, 2018

@thw0rted, that's actually a really good point and may be a good reason to update Cesium to number instead of Number.

@bampakoa
Copy link
Contributor

@szechyjs I have used your typings in one of my past projects and looks awesome. Very nice effort!

@mramato thank you very much for your feedback, I really appreciate. I am currently in the process of fixing bugs and contribute back to the library. As soon as, I have a good working copy, I will get back with a PR so anyone can take a look and help.

@bampakoa
Copy link
Contributor

In the latest releases, there are changes in the documentation of the source code that allow extraction of typings file using tsd-jsdoc. I would ❤️ to give more details in case anyone is interested.

@hpinkos @pjcozzi @mramato do you think it would be useful to write a blog post for that?

@emackey
Copy link
Contributor

emackey commented Mar 14, 2019

Related, can there be / should there be a build step for generating typings?

@emackey
Copy link
Contributor

emackey commented Mar 29, 2019

@bampakoa Were you able to get to a state where the typings file came directly from tsd-jsdoc, with no manual edits?

@bampakoa
Copy link
Contributor

@emackey yes, I was able to generate a typings definition file using tsd-jsdoc but I have not verified its correctness and quality yet. It would be great if anyone could review that.

@bampakoa
Copy link
Contributor

bampakoa commented Oct 7, 2019

Hey all! Today Microsoft announced Typescript 3.7 Beta which contains a significant change as far as typings generation is concerned.

--declaration and --allowJs options will finally play nice together.

This actually means that Typescript will do its best to extract type definitions from JSDoc statements by its own and the use of 3rd party tools such as tsd-jsdoc may not be needed in the future 🎉.

This also means that Cesium may be able to publish its own types embedded into the library 😃

@puckey
Copy link
Contributor

puckey commented Oct 18, 2019

I have successfully generated typescript types for the source files using the latest Typescript 3.7 beta: tsc --declaration --allowJs --emitDeclarationOnly ./Source/Cesium.js

@thw0rted
Copy link
Contributor Author

Would you mind throwing those up e.g. in a gist somewhere? I'd like to try it on my project and see if it still compiles, vs. the hand-groomed ones I've been using so far.

@puckey
Copy link
Contributor

puckey commented Oct 18, 2019

Here you go: https://www.dropbox.com/s/wih9qtxn2e4x642/cesium-typescript-types.zip?dl=1

These are generated by downloading the Cesium repo as zip, installing and building (to create /Source/Cesium.js and afterwards:

yarn add typescript@beta
npx tsc --declaration --allowJs --emitDeclarationOnly --outDir ./types ./Source/Cesium.js

I am unable to generate them as a single .d.ts file, because some of the files are giving errors during building: https://gist.github.com/puckey/d333f8f7c51ef80ed962694f1e4f7517

I have also noticed that some of the types are missing. For example Scene#camera is missing and FeatureDetection.d.ts is empty..

@thw0rted
Copy link
Contributor Author

Ah, OK. Well, thanks for the upload and I might try to poke at it, but if it's not "drop in ready" yet then I'll set it aside until I have a little more time. Ideally, I could make the typings, fix the JSDoc where stuff is missing / wrong, and iterate until at least my own code builds cleanly. I'll try to get to it when I have the time, for sure.

@puckey
Copy link
Contributor

puckey commented Oct 18, 2019

It seems that the missing types are related to Typescript not yet supporting @memberof, which is used in all the cesium getters / setters: microsoft/TypeScript#28730

Overview of supported jsdoc tags: https://github.com/microsoft/TypeScript-wiki/blob/master/JSDoc-support-in-JavaScript.md

@puckey
Copy link
Contributor

puckey commented Oct 18, 2019

Looking into some of the jsdocs annotations, I am noticing inconsistencies like the following in Scene.prototype.mapProjection, where the type is set as MapProjection (which does not exist) while the default references GeographicProjection:

        /**
         * Get the map projection to use in 2D and Columbus View modes.
         * @memberof Scene.prototype
         *
         * @type {MapProjection}
         * @readonly
         *
         * @default new GeographicProjection()
         */

Leads me to wonder if generating types from comments like this would ever lead to a correct set of typings.. Comments are notoriously hard to keep up to date.

@thw0rted
Copy link
Contributor Author

The Typing generation step would need to be part of CI, if not maybe also the local build tooling (rollup etc). We need to fix all the comments once, then instrument so that future failings break the build loudly and block PR merging.

@puckey
Copy link
Contributor

puckey commented Oct 18, 2019

(I will refrain from proposing Typescript as the solution) : )

@thw0rted
Copy link
Contributor Author

Too late! We can just insidiously make it easier and remove friction points until whoops, look at that, the code is basically TS already 😁

@thw0rted
Copy link
Contributor Author

OK, I had a little time to look at this today. My goal is to eventually stop having to hand-maintain one giant cesium.d.ts and have generated typings on a per-module level as well as a top level one that matches the (generated) Source/Cesium.js.

I'm walking out the door now, but if you want to play with my first pass, it at least builds (npm run build-typings). A number of tests fail in the Jasmine runner, but I didn't think to try running it beforehand so I'm not sure if they were my fault. I'm re-testing now but I'm going to have to see what (if anything) failed when I get back on Monday.

@thw0rted
Copy link
Contributor Author

I went back and re-ran the tests. Three are still failing (c.f. #7892) but that's the same before and after my changes. I'm looking to see if there are any alternatives to @memberof support because I don't see that showing up any time soon. I think it might be possible to convert the legacy pattern to proper ES6 classes without breaking the API but that's a heck of a lot of work.

Will update the issue if I get any clever new ideas.

mramato added a commit that referenced this issue May 27, 2020
It's been a long requested feature for us to have official TypeScript type
definitions.  While the community has done a yeoman's job of manually
supporting various efforts, the most recent incarnation of which is
`@types/cesium`, the sheer scale and ever-evolving nature of Cesium's code
base makes manual maintenance a Sisyphean task.

Thankfully, our decision to maintain meticulous JSDoc API documentation
continues to pay dividends and is what makes automatically generating
TypeScript definitions possible. Using the excellent
https://github.com/englercj/tsd-jsdoc project we can now automatically
generate and even partially validate official definitions as part of the
build process. (Thanks to @bampakoa who contributed some early PRs to both
CesiumJS and tsd-jsdoc over a year ago and is how I learned about
tsd-jsdoc)

While tsd-jsdoc output is mostly where we need it to be, we do
post-processing on it as well. This lets us clean up the output and also
make sure these definitions work whether users include cesium via module,
i.e. `import { Cartesian3 } from 'cesium'`, or individual files, i.e.
`'import Cartesian3 from 'cesium/Source/Core/Cartesian3'`. There were also
some quirks of tsd-jsdoc output we fixed that may eventually turn into a PR
into that project from us. The post-processing is part typescript compiler
API, part string manipulation. It works and is straightforward but we might
want to go full TS api in the future if we decide we need to do more
complicated tasks. The output of tsd-jsdoc is currently a little noisy
because of some incorrect error reporting, but I'm talking with the
maintainer in englercj/tsd-jsdoc#133 to get them
fixed. No need to hold up this PR for it though.

The definition is generated as a single `Cesium.d.ts` file in Source, so it
lives alongside Cesium.js. It is ignored by git but generated by a separate
`build-ts` task as part of CI and makeZipFile. This task also validates the
file by compiling it with TypeScript, so if a developer does anything too
egregious, the build will fail. Definitions are automatically included in
our npm packages and release zips and will be automatically used by IDEs
thanks to the `types` setting in package.json. This means that IDEs such as
VS Code will prefer these types over the existing `@types/cesium` version
by default.

I didn't want to slow the `build` step down, so I made this a separate
step, but in the future we could consider running it by default and we
could also unignore this file in Git so that PR reviewers can see the
impact, if any, our code changes have on the generated definitions. This
might be a good idea as an additional sanity check and should only actually
change when the public API itself changes. But the issue would be
remembering to run it before submitting the code (or we could use git hooks
I suppose?) I just don't want to slow down devs so I'm hesitant to do
anything like this out of the gate. We can definitely revisit in the
future.

A particular exciting thing about this approach is that it exposed a ton of
badness in our current JSDoc markup, which is now fixed. Previously, our
only concern was "does the doc look right" and we didn't pay attention to
whether the meta information generated by JSDoc correctly captured type
information (because up until it didn't matter). We leaned particular hard
on `@exports` which acted as a catch-all but has now been completely
removed from the codebase. All this means is that our documentation as a
whole has now improved greatly and will continue to be maintained at this
new higher level thanks to incorporating TS definition creation into our
pipeline!

One minor caveat here is that obviously we changed our JSDoc usage to both
make it correct and also accommodate TypeScript. The main drawback to these
fixes is that enums are now generated as globals in the doc, rather than
modules. This means they no longer have their own dedicated page and are
instead on the globals page, but I changed the code to ensure they are
still in the table of contents that we generate. I think this trade-off is
perfectly fine, but I wanted to mention it since it does change the doc
some. We can certainly look into whether we can generate enums on their own
page if we think that makes sense. (I actually like this approach a little
better personally).

Last major piece, the actual code. 99% of the changes in this PR only
affect the JSDoc. There are two exceptions:

A few of our enums also have private functions tacked onto them. I had to
move these functions to be outside the initializer but otherwise they are
unchanged.  This ensures that a valid TS enum is generated from our code, since you can't have functions globbed onto enums in the TS world. If we were writing TS code by hand, we could use  declaration merging with a namespace, but the toolchain we are using doesn't have a way to express that right now.  There were two cases where these extra functions weren't private, `ComponentDataType` and `IndexDataType`. That means that as far as the TS definitions goes, the helper methods don't exist.  I consder this an edge case and we can write up issues to investigate later.  I'm actually not even sure if these functions are public on purposes, @lilleyse can you confirm?

We had a few places where we had method signatures with optional parameters
that came _before_ required parameters, which is silly. This is invalid
TypeScript (and not good API design no matter the language). In 99% of
cases this was `equalsEpsilon` style functions where the lhs/rhs were
optional but the epsilon was not. I remember the discussion around this
when we first did it because we were paranoid about defaulting to 0, but
it's an edge case and it's silly so I just changed the epsilon functions
to default to zero now, problem solved.

Here's a high level summary of the JS changes:

* Use proper `@enum` notation instead of `@exports` for enums.

* Use proper `@namespace` instead of `@exports` for static classes.

* Use proper `@function` instead of `@exports` for standalone functions.

* Fix `Promise` markup to actually include the type in all cases, i.e.
`Promise` => `Promise<void>` or `Promise<Cartesian3[]>`.

* Fix bad markup that referenced types that do not exist (or no longer
exist) at the spec level, `Image` => `HTMLImageElement`,
`Canvas` => `HTMLCanvasElement`, etc.. `TypedArray` in particular does not
exist and much be expressed as a lsit of all applicable types,
`Int8Array|Uint8Array|Int16Array|Uint16Array...`.

* Use dot notation instead of tilde in callbacks, to better support
TypeScript, i.e. `EasingFunction~Callback` becomes
`EasingFunction.Callback`. The only exception is when we had a callback
type that was i.e. `exportKml~ModelCallback` becomes
`exportKmlModelCallback` (a module global type rather than a type on
exportKml). This is because it's not possible to have exportKml be both a
function and a namespace in this current toolchain.  Not a big deal either
way since these are meta-types used for defining callbacks but I wanted to
mention it.

* There were some edge cases where private types that were referenced in
the public API but don't exist in the JSDoc. These were trivial to fix by
either tweaking the JSDoc to avoid leaking the type or in some cases, just
as `PixelDataType`, simply exposing the private type as public.  I also
found a few cases where things were accidentally public, I marked these as
private (these were extreme edge cases so I'm not concerned about breaking
changes). Appearances took an optional `RenderState` in their options, I
just changed the type to `Object` which we can clean up further later if
we need to.

* Lots of other little misc JSDoc issues that became obvious once we
started to generate definitions (duplicate parameters for example).

Thanks again to the community for helping generate ideas and discussions
around TS definitions over the last few years and a big thanks to @javagl
for helping behind the scenes on this specific effort by evaluating a few
different approaches and workaround before we settled on this one (I'm
working on a blog with all of the gory details for those interested).

Finally, while I'm thrilled with how this turned out (all ~41000 lines
and 1.9MB of it), I can guarantee we will uncover some issues with the
type definitions as more people use it. The good news is that improving it
is now just a matter of fixing the JSDoc, which will benefit the community
as a whole and not just TS users.


Fixes #2730
Fixes #5717
@mramato
Copy link
Contributor

mramato commented May 27, 2020

For anyone following this issue, check out #8878. Your feedback is appreciated.

@cesium-concierge
Copy link

Congratulations on closing the issue! I found these Cesium forum links in the comments above:

https://groups.google.com/forum/#!topic/cesium-dev/25jPqr5ODg0
https://groups.google.com/forum/#!searchin/cesium-dev/Angular%7Csort

If this issue affects any of these threads, please post a comment like the following:

The issue at #5717 has just been closed and may resolve your issue. Look for the change in the next stable release of Cesium or get it now in the master branch on GitHub https://github.com/AnalyticalGraphicsInc/cesium.

katSchmid added a commit to PropellerAero/cesium that referenced this issue Jun 24, 2020
* Add translucency rectangle to sandcastle example [skip ci]

* Account for primitive restart value.

* No const in ES5.

* Simplify skinning to always expect VEC4 joint/weight accessors.

* Changed code mirror viewport margin to Infinity for improved search

* Moves changes to 1.70

* Remove unused imports.

* Fix variable and semantic names.

* Added frustumSplits option to DebugCameraPrimitive

* updated CHANGES.md

* Added GlobeTranslucencyState

* Fixed ground primitives in orthographic mode

* Changed czm_writeDepthClampedToFarPlane to czm_writeDepthClamp and czm_depthClampFarPlane to czm_depthClamp

* Add cjs extension to prettier configuration

I was modifying gulpfile in another branch and realized formatting was
not set up correctly.

* Fixed depth plane for orthographic

* updated CHANGES.md

* Made orthographic depth plane aligned to camera and fixed multifrustum ray ellipsoid intersection

* made common variable for camera

* Fixed derived commands

* Remove invisible translucency state to support classification on invisible surfaces

* Fix picking in 2D

* Use better image for sandcastle [skip ci]

* Don't do depth only shader if clipping planes are enabled [skip ci]

* Updated unit tests

* Fix a typo

* Fix a typo

* Render full sky atmosphere when globe is hidden

* Add per-fragment atmosphere, improve per-vertex atmosphere

* Update CHANGES.md

* Fix tests

* Add underground color

* Add getters/setters, Sandcastle, tests

* Update CHANGES.md

* Fix CHANGES.md [skip ci]

* Added nightAlpha and dayAlpha in the ImageryLayer

* Updated CONTRIBUTORS.md

* CHANGES.md updated

* Added night textures showcase to the Sandcastle

* typo

* Switch to two-pass classification

* Updates from review

* Tweak atmosphere under ellipsoid so it's less bright

* Fix tests

* removed old comment [skip ci]

* Distance relative to globe surface

* Rename undergroundColorByDistance to undergroundColorAlphaByDistance

* Updates from review

* Doc [skip ci]

* Render underground when clipping planes or cartographic limit are being used

* Add czm_backFacing

* Only render ground atmosphere for front faces

* Show back faces in cartogrphic limit rectangle sandcastle

* Fix camera height uniform

* Temp

* Align ellipsoid geometry to camera

* Default perFragmentAtmosphere to false

* Support #rgba and #rrggbbaa in Color.fromCssColorString

* Updates from review

* Same blueness everywhere

* Improve alpha

* Tweak brightness by fade distance

* Revert ground atmosphere change to fix issues when zoomed out really far

* Updates from review

* Remove old intersection code

* Fix lighting crash

* Tweak strength to prevent ring artifact in enableLighting mode

* Turn off globe lighting in demo

* Reorder ifdef's

* Reorder updateFrameState

* Added GlobeTranslucency to group translucency options

* Fix test

* Remove frameState from update function

* Premultiply alpha for ground polylines

* Fix tests

* Fix translucencyEnabled not applying when base color alpha is < 1.0

* Define constant once

* Add back translucency gradient to sky atmosphere

* Add define for globe translucency to SkyAtmosphereCommon

* Sky atmosphere fix to not use camera height when intersecting the ellipsoid

* some tweaks to day/night blending

* Generate official TypeScript type definitions

It's been a long requested feature for us to have official TypeScript type
definitions.  While the community has done a yeoman's job of manually
supporting various efforts, the most recent incarnation of which is
`@types/cesium`, the sheer scale and ever-evolving nature of Cesium's code
base makes manual maintenance a Sisyphean task.

Thankfully, our decision to maintain meticulous JSDoc API documentation
continues to pay dividends and is what makes automatically generating
TypeScript definitions possible. Using the excellent
https://github.com/englercj/tsd-jsdoc project we can now automatically
generate and even partially validate official definitions as part of the
build process. (Thanks to @bampakoa who contributed some early PRs to both
CesiumJS and tsd-jsdoc over a year ago and is how I learned about
tsd-jsdoc)

While tsd-jsdoc output is mostly where we need it to be, we do
post-processing on it as well. This lets us clean up the output and also
make sure these definitions work whether users include cesium via module,
i.e. `import { Cartesian3 } from 'cesium'`, or individual files, i.e.
`'import Cartesian3 from 'cesium/Source/Core/Cartesian3'`. There were also
some quirks of tsd-jsdoc output we fixed that may eventually turn into a PR
into that project from us. The post-processing is part typescript compiler
API, part string manipulation. It works and is straightforward but we might
want to go full TS api in the future if we decide we need to do more
complicated tasks. The output of tsd-jsdoc is currently a little noisy
because of some incorrect error reporting, but I'm talking with the
maintainer in englercj/tsd-jsdoc#133 to get them
fixed. No need to hold up this PR for it though.

The definition is generated as a single `Cesium.d.ts` file in Source, so it
lives alongside Cesium.js. It is ignored by git but generated by a separate
`build-ts` task as part of CI and makeZipFile. This task also validates the
file by compiling it with TypeScript, so if a developer does anything too
egregious, the build will fail. Definitions are automatically included in
our npm packages and release zips and will be automatically used by IDEs
thanks to the `types` setting in package.json. This means that IDEs such as
VS Code will prefer these types over the existing `@types/cesium` version
by default.

I didn't want to slow the `build` step down, so I made this a separate
step, but in the future we could consider running it by default and we
could also unignore this file in Git so that PR reviewers can see the
impact, if any, our code changes have on the generated definitions. This
might be a good idea as an additional sanity check and should only actually
change when the public API itself changes. But the issue would be
remembering to run it before submitting the code (or we could use git hooks
I suppose?) I just don't want to slow down devs so I'm hesitant to do
anything like this out of the gate. We can definitely revisit in the
future.

A particular exciting thing about this approach is that it exposed a ton of
badness in our current JSDoc markup, which is now fixed. Previously, our
only concern was "does the doc look right" and we didn't pay attention to
whether the meta information generated by JSDoc correctly captured type
information (because up until it didn't matter). We leaned particular hard
on `@exports` which acted as a catch-all but has now been completely
removed from the codebase. All this means is that our documentation as a
whole has now improved greatly and will continue to be maintained at this
new higher level thanks to incorporating TS definition creation into our
pipeline!

One minor caveat here is that obviously we changed our JSDoc usage to both
make it correct and also accommodate TypeScript. The main drawback to these
fixes is that enums are now generated as globals in the doc, rather than
modules. This means they no longer have their own dedicated page and are
instead on the globals page, but I changed the code to ensure they are
still in the table of contents that we generate. I think this trade-off is
perfectly fine, but I wanted to mention it since it does change the doc
some. We can certainly look into whether we can generate enums on their own
page if we think that makes sense. (I actually like this approach a little
better personally).

Last major piece, the actual code. 99% of the changes in this PR only
affect the JSDoc. There are two exceptions:

A few of our enums also have private functions tacked onto them. I had to
move these functions to be outside the initializer but otherwise they are
unchanged.  This ensures that a valid TS enum is generated from our code, since you can't have functions globbed onto enums in the TS world. If we were writing TS code by hand, we could use  declaration merging with a namespace, but the toolchain we are using doesn't have a way to express that right now.  There were two cases where these extra functions weren't private, `ComponentDataType` and `IndexDataType`. That means that as far as the TS definitions goes, the helper methods don't exist.  I consder this an edge case and we can write up issues to investigate later.  I'm actually not even sure if these functions are public on purposes, @lilleyse can you confirm?

We had a few places where we had method signatures with optional parameters
that came _before_ required parameters, which is silly. This is invalid
TypeScript (and not good API design no matter the language). In 99% of
cases this was `equalsEpsilon` style functions where the lhs/rhs were
optional but the epsilon was not. I remember the discussion around this
when we first did it because we were paranoid about defaulting to 0, but
it's an edge case and it's silly so I just changed the epsilon functions
to default to zero now, problem solved.

Here's a high level summary of the JS changes:

* Use proper `@enum` notation instead of `@exports` for enums.

* Use proper `@namespace` instead of `@exports` for static classes.

* Use proper `@function` instead of `@exports` for standalone functions.

* Fix `Promise` markup to actually include the type in all cases, i.e.
`Promise` => `Promise<void>` or `Promise<Cartesian3[]>`.

* Fix bad markup that referenced types that do not exist (or no longer
exist) at the spec level, `Image` => `HTMLImageElement`,
`Canvas` => `HTMLCanvasElement`, etc.. `TypedArray` in particular does not
exist and much be expressed as a lsit of all applicable types,
`Int8Array|Uint8Array|Int16Array|Uint16Array...`.

* Use dot notation instead of tilde in callbacks, to better support
TypeScript, i.e. `EasingFunction~Callback` becomes
`EasingFunction.Callback`. The only exception is when we had a callback
type that was i.e. `exportKml~ModelCallback` becomes
`exportKmlModelCallback` (a module global type rather than a type on
exportKml). This is because it's not possible to have exportKml be both a
function and a namespace in this current toolchain.  Not a big deal either
way since these are meta-types used for defining callbacks but I wanted to
mention it.

* There were some edge cases where private types that were referenced in
the public API but don't exist in the JSDoc. These were trivial to fix by
either tweaking the JSDoc to avoid leaking the type or in some cases, just
as `PixelDataType`, simply exposing the private type as public.  I also
found a few cases where things were accidentally public, I marked these as
private (these were extreme edge cases so I'm not concerned about breaking
changes). Appearances took an optional `RenderState` in their options, I
just changed the type to `Object` which we can clean up further later if
we need to.

* Lots of other little misc JSDoc issues that became obvious once we
started to generate definitions (duplicate parameters for example).

Thanks again to the community for helping generate ideas and discussions
around TS definitions over the last few years and a big thanks to @javagl
for helping behind the scenes on this specific effort by evaluating a few
different approaches and workaround before we settled on this one (I'm
working on a blog with all of the gory details for those interested).

Finally, while I'm thrilled with how this turned out (all ~41000 lines
and 1.9MB of it), I can guarantee we will uncover some issues with the
type definitions as more people use it. The good news is that improving it
is now just a matter of fixing the JSDoc, which will benefit the community
as a whole and not just TS users.


Fixes CesiumGS#2730
Fixes CesiumGS#5717

* Don't hardcode ./node_modules/tsd-jsdoc

It may be different in some environments. Instead, just use the normal module name, which will be resolved according to node's module resolution rules.

* Use tsconfig.json to avoid errors in some environments.

* Add new Sandcastle example.

* Add back a "Proxy" type.

Since master documented a `Proxy` base class, create it rather than using
`DefaultProxy` everywhere.

* Add missing `update` function to all DataSource implementations

Allows them to be assignable to `DataSource` in the TS definitions.

* Minor tweak to doc and Sandcastle

* JSDoc fixes to GregorianDate and TimeIntervalCollection

* Fix JSDoc for exportKml

* Additional JSDoc/TypeScript fixes.

* Make IonImageryProvider actually implement all ImageryProvider properties

* More TypeScript fixes

1. Make Matrix 2/3/4 ArrayLike objects

2. Make PropertyBag include a string indexor so that it is a dictionary
like object. We do this by making up a new private interface and defining
it only in the definition file. JSDoc ignores it in the HTML output.

* Update CHANGES

Fix bad commit.

* Number[] -> number[] in TS definition

Add workaround for englercj/tsd-jsdoc#117

* Fix Event.raiseEvent TS generation.

* Add missing availability property to TerrainProvider classes.

* Whoops

* More JSDoc fixes and TS improvements

1. Resource has incorrect return types.
2. Improve formatting of TS definition file.

* Don't abuse EllipsoidTerrainProvider.

* Lots of small changes to improve the typescript definitions.

* Give up on trying JSDoc tags for PropertyBag

* Example of exporting ConstructorOptions interface

* Break out constructor options: Entity and Graphics

* Various small type fixes

* Selected/tracked entity can be undefined

* First pass at generic Events

* Constructor interfaces for ImageryProviders

* Load-method option interfaces for some DataSources

* Options interfaces for Viewer, Resource

* CesiumGS#8843 Clear Request cancelFunction and deferred references to free up unintentional retained request data memory

* CesiumGS#8843 Add comments

* CesiumGS#8843 Update comment

* Add CONTRIBUTORS entry

* Use function overloading for exportKml

* Update CHANGES.md

* Update CesiumSandcastle.js

* Spell out docs for Resource.QueryParameter

* Undo some changes

* Add ConstructorOptions to the rest of the ImageryProviders

* Workaround CesiumMath -> Math documentation issue

* Fix doc links that include a hash.

* Improve camera controls when underground

* Update CHANGES, tweak doc, and add test.

* Fix UrlTemplateImageryProvider

Also opt-in tsd-jsdoc so we can use @template in the future.

* More fixes found by Kevin.

* Tweak CHANGES related to TypeScript

* Remove hardcoded node_modules in gulpfile.

1. Use `npx` to execute binaries rather than hardcoded `node_modules/.bin`,
which may not always be the right location.

2. Streamline generateDocumentation by calling it sync.

* Update CHANGES to make underground visualization more prominent

* Fix doc using @member instead of @memberof

* Removed duplicate CHANGES.md line

* Add Cesium OSM Buildings note in CHANGES

* Add Sandcastle link [ci skip]

* Fix typo [ci skip]

* fixed flipy issue

* updated CHANGES.md

* added myself to CONTRIBUTORS.md

* added tests for flipY function

* Fixed Bing Maps Road option in sandcastle example

* removed extra whitespace line

* fix gulp build error

* Added Navagis to CONTRIBUTORS.md

* updated tokens, version, and changes.md

* Fix variable naming in fog sandcastle

* Add toString method for Resource class

* Tests for Resource#toString

* Use explicit cast

* Improve JSDoc/TypeScript support for Material properties

All of the MaterialProperty constructors take primitive types in addition
to Property instances (and automatically wrap the primitive type in a
`ConstantProperty`).  The JSDoc was incorrect on this leading to sub-par
TS type definitions. Also fixed a missing optional result parameter in
the JSDoc for `EllipsoidGeodesic.interpolateUsingSurfaceDistance`.

Fixes CesiumGS#8898

* Update CHANGES

Add another missing optional result fix.

* Update CHANGES for cancelled request memory leak fix

* Add additional smokescreen to build-ts

1. Add a Specs/TypeScript directory with a minimal TS configuration that
uses the d.ts files generated by build-ts

2. Have `build-ts` compile index.ts from this directory which makes sure
various types actually conform to their expected interfaces.

3. Fix ImageryProvider interfaces which had issues exposed by this new
test.

In the future we can add any additional smokescreens that we think are
important to validate our definitions going forward, but this will never
be a fully exhaustive check.

We currently don't actually execute the output, it's just there for
compile-time checking.  We can revisit this if we ever feel that it's
needed.

* Update CHANGES

* Add missing property typs to TypeScript smokescreen

* Other minor TS improvements.

* Make sure all nullable Entity API properties are marked as such.

* Most ImageryProvider properties can be undefined.

* Fix missing HeadingPitchRoll/HeadingPitchRange JSDoc

* Missing `|undefined` in GoogleEarthEnterpriseMapsProvider

* Missing JSDoc

* change cesiumScriptRegex value

fixes Issue CesiumGS#8902

* Expose buildModuleUrl as part of the Cesium API

It's been used in Sandcastle examples forever, but has always been marked
as private in jsdoc. In order to make it available in TypeScript, we need
to expose it.

Also cleaned up the doc to be more inline with what the current version
actually does.

Fixes CesiumGS#8922

* add more test

add CesiumSomething test

* Correcting Navagis Corporate Link

* Two small JSDoc/TS fixes

1. `EllipsoidTangentPlane.fromPoints` takes an array.
2. `EntityCollection.getById` and `CompositeEntityCollection.getById` can
both return undefined.

* Fix whitespace

* Update CHANGES.

Also fix missing links from previous PRs

* Add some missing readonly documentation tags

* Add false as a possible value for skyBox and SkyAtmosphere

* CesiumGS#8927: Exposing Transforms.rotationMatrixFromPositionVelocity; Adding tests for Transforms.rotationMatrixFromPositionVelocity

* CesiumGS#8927: Adding message for exposing Transforms.rotationMatrixFromPositionVelocity to CHANGES.md

* CesiumGS#8927: Running prettier on new rotationMatrixFromPositionVelocity tests

* Only include Source folder in TypeScript smokescreen tests.

We were including all Cesium directories in the TypeScript smokescreen
configuration. This meant that multiple Cesium.d.ts files were included
if you did a full build and then ran `build-ts` afterwards. This updates
the config used by the smokescreen to only include Source.

* Improve base url regex, add additional test.

* CesiumGS#8927: Simplifed rotationMatrixFromPositionVelocity tests

* Fix geometry creation in TypeScript

None of the `XXXGeometry` classes are actually `Geometry` instances. They
are instead utility classes that create geometries via their
`createGeometry` implementation. `GeometryInstance` can take either "type"
but since JS doesn't have types we never really defined what the "utility"
type is, so the TypeScript definition for `GeometryInstance` specifies that
currently only specifies `Geometry`. This means that this valid JS code
is a compiler error in TypeScript

```
const geometryInstance = new GeometryInstance({
  geometry: new PolylineGeometry({
    positions: [],
  }),
});
```

To fix this, I introduced a `GeometryFactory` base class like we have
elsewhere in the code and changed `GeometryInstance` to take either type.
This is the only place where we actually base "non-geometry Geometry" in
the API.

Happy to consider other names, like `GeometryCreator` if we don't like
factory for some reason, but I want to get this in sooner rather than
later for 1.70.1 fixes.

Also fixed an issue with tsconfig.json I introduced in my last change
which was failing to actually catch TS compile errors because it wasn't
including the Cesium.d.ts.

* Update doc/CHANGES

* Bump version and cleanup CHANGES for 1.70.1 release.

* adding release 1.70.1

* add codeship

* make tests runable

* adding propeller code changes back

* more changes form diff

* Correcting ChANGES.md and index.d.ts

Co-authored-by: Matthew Amato <[email protected]>
Co-authored-by: Sean Lilley <[email protected]>
Co-authored-by: Kevin Ring <[email protected]>
Co-authored-by: Ed Mackey <[email protected]>
Co-authored-by: Ian Lilley <[email protected]>
Co-authored-by: Sam Suhag <[email protected]>
Co-authored-by: Sanjeet Suhag <[email protected]>
Co-authored-by: Jakub Vrána <[email protected]>
Co-authored-by: Edvinas Pranka <[email protected]>
Co-authored-by: Omar Shehata <[email protected]>
Co-authored-by: Omar Shehata <[email protected]>
Co-authored-by: James Bromwell <[email protected]>
Co-authored-by: Brandon Nguyen <[email protected]>
Co-authored-by: Hannah <[email protected]>
Co-authored-by: Eli Bogomolny <[email protected]>
Co-authored-by: Peter Gagliardi <[email protected]>
Co-authored-by: Jonathan Nogueira <[email protected]>
Co-authored-by: xiaobaogeit <[email protected]>
Co-authored-by: Jonathan Nogueira <[email protected]>
Co-authored-by: Frederic Junod <[email protected]>
Co-authored-by: John Remsberg <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.