-
Notifications
You must be signed in to change notification settings - Fork 954
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Getting Turf back on track #1428
Comments
Hi @tmcw Thanks for taking the time to check in on the status of the project, this has certainly been weighing on my mind over the past couple of months. That said I want to start by saying the following
Re Typescript
On a personal & professional note I have no desire or need to learn TypeScript, TurfJS would be my sole reason for learning it. Re Lerna
Contributing.md Re All-in on ES6
Could you elaborate on use case 1 ('Script-tag or browserify users') - is that basically folks using Turf via a CDN? Why would that be slower under this proposal? I think I cautiously support this approach. The bulk of browser apps (at least serious ones) are being built with some sort of bundling tool so I dont think we'd be leaving many out in the cold with this decision. My one proviso is that I think it would be nice to have some sort of basic bundled version (eg a CDN version) for people to hack with and learn with that doesn't require them to get into the whole world of bundlers, it wouldn't be the preferred method of usage but would be there to provide a low barrier of entry. Anyway those are my initial thoughts - will ponder further and see if there is anything else that comes to mind. Cheers |
If we were to publish a single ES6 module, then users who go through Rollup or Webpack would be able to do:
And their application bundle would only include the code for the But users who go through browserify, who would write
Would get the full turf bundle, including JSTS, regardless of how much or little of the API surface they use. And, well - now that I look at it, it looks like there aren't UMD bundles for individual parts of turf, so people using the CDN route would have the same experience: they'd always load the full library. So the only real drawbacks are:
|
https://github.com/lerna/lerna-changelog would be another option, which generates the changelog from PR titles instead of commits
this seems like a good solution, but it would need to make sure that subdependencies are only imported by those functions that actually need them. e.g. the
I'm not sure I understand why you want to remove TypeScript. It usually makes the APIs much clearer for users, even if there is no immediate advantage to the development of this project itself. External typings are an alternative but they tend to get out of sync and usually just don't work as well. Since TypeScript is mostly just a superset of JS it should work fine too for files without any type annotations, so even the current JS-only packages could be compiled with Assuming things like #1297 can be resolved I'd prefer to keep/make TypeScript part of the project. I had mentioned before that I would be interested in helping move the project to TS, but I agree that in the current state it is hard to contribute to. |
This isn't a current issue, right? For example, the point helper doesn't import JSTS.
The reasons are pretty clearly stated in the OP - we have very few contributors and they aren't TypeScript masters. Removing typescript would create a path to a once-again-deployable Turf. Given more contributors, and contributors that knew TS inside and out, then TS would be a good idea, but we build projects with the contributors we have, not the ones we want. In other words, there is a disadvantage of TS to the development of the project itself - it's slowing down development. If you want to push the TS move to completion, then - well - that'd be very welcome! I'm just reading the context, that the TypeScript migration kicked off in January, it's about halfway finished, and the main contributor of the TS parts isn't very active, so - given those priors, it's hard to anticipate a significant change. |
no, it's not. but if all implementations at the end were in a single file with a top level import it could become a problem, although given the amount of packages in this project it seems quite unlikely to be the endresult :)
sounds like a chicken-and-egg problem to me. if the project wasn't split into so many small packages, each with their own TS integration need, it would would be much easier to use TS for the whole project. and as I mentioned before, once TS is setup properly it can support both TS and JS and I'm assuming to support Node.js we would need something that compiles also to CommonJS anyway, right? |
Yeah, to be clear - 'a single ES6 module' means a single module published to npm, with a single entry point. It would not mean that all the code is in one file.
Yes, regardless of the path we take, we'd need to create a CommonJS entry point; that need doesn't go away unless we switch everything back to require(), which I don't think anyone wants to do. That step in the process isn't (imho) really a deciding factor - creating CJS entry points works with rollup, works with typescript, and plenty of other systems. If that were the main need, the most obvious solution would be something like Rollup, that we'll need to use anyway for creating a UMD bundle. |
Thanks for starting this issue. For my part, I'm looking forward to contributing more. I don't have very much experience in packaging modules, and certainly not enough to make architectural decisions. But once we have a direction and a bunch of modules that need updating (in which ever direction) I'm happy to help with grunt work. FWIW my perspectives on the key decisions: TypescriptI was definitely not thrilled about using it, and it feels like a big barrier to any new contributor. (I suspect the actual barrier is less than the perceived one, based on my tiny experience, but that's a bit beside the point). It seems like the potential gains were fairly modest, the actual gains have been almost non existent, and the downsides are significant, so...pretty easy decision. Single-module ES6Personally, I still use Browserify, because it's what the dev tool stack I use uses, and I haven't had a compelling reason to switch to Webpack. I'm probably far from alone. So yeah, I'd be worse off. I'm not sure I'd even heard of "tree shaking" until this thread. But honestly, the maintainability of the whole project is a much bigger consideration than bundle size for a subset of users. We should probably just make sure we alert potential users to this fact and maybe give them some useful pointers. At the moment, it looks like the complete Turf bundle (via Browserify) is 557k. That's actually not that crazy. Still fine IMHO for people who are newbie web developers to just import the whole thing via a script tag. (I used to work on a website that shipped I think 7MB of JS at first page load so...yeah). |
Okay, well, if folks agree on those points, the path forward would be moving everything from Consider this an open task, if there's consensus on the way forward (on either part, the TS removal, single module, lerna arrangement, or just uniformity). My personal time, priorities, and interests don't line up here, so I'm not going to be contributing code or rejoining as a maintainer. |
I'm ok with removing TypeScript, cause for me as well it's kinda a barrier at the moment. Regarding the structure, I trust @tmcw that a single ES6 module would drastically reduce complexity, and for ES6 modules I agree with @stevage, importing the whole library I don't think is a big deal. However, to solve the import issue I was wondering if it was possible to automatically generate separate individual compiled files for each package/function, so that who wants/needs to use require or load on the browser to use one or more Turf functions could do that. import {pointGrid} from 'turf';
var pointGrid = require('turf/require/point-grid');
<script src="cdn/to/turf/scripts/point-grid"></script> Is the above possible, if it even makes sense? |
@stebogit FWIW in theory it should be possible to still generate those single-module packages using e.g. rollup with generated entry files. I'm just not sure if that's worth the effort... |
That isn't correct: if you're using browserify or nodejs, and you sub-require a commonjs file within a package, like |
@tmcw @stebogit @rowanwins I know I've been away for a long time, I've deep dived into the blockchain rabbit hole. I'm 👍 if we drop Typescript, it was actually harder then I expected to convert the code base. I can help refactor Typescript out of it, we should keep it ES6+ that way it works well with Rollup. I'm 👍 if we convert it to a monorepo |
So it sounds like we've got consensus on a desired way forward - thanks for the suggestions @tmcw and others for chiming in their thoughts. I'm happy to take the lead on a pull request outlining the revised repo structure along the lines of what @tmcw has suggested above. I'll also include a couple of worked examples for modules. Hopefully once we've got the pattern sorted we might be able to work together to get some of the changes made at a steady pace. I'll try and aim to get something up in the next few days so stay tuned. Given that this will be a substantial change we'll be releasing this as a new major version so this would be a good time to tackle any other breaking changes, for example addressing this one comes to mind. |
I agree with dropping TypeScript, it's a big barrier to entry for new contributors (one extra thing to learn).
That would be nice, as a developer using turf, a big annoyance is knowing which module a method is in. Is it in helpers, it's own module or meta... I'm constantly needing to check the docs on this. I just want to import whatever methods I need from |
Amen @andrewharvey! 😄 |
+1 Amen! |
Is my understanding we are going to build the final bundle using Rollup (or some other bundler), correct? We don't need to convert the entire codebase now, that can be done bit by bit at any time or just if/when a module needs to be fixed or modified. But since we are restructuring the project I suppose this would be a good time to introduce that upgrade as well. |
The main Turf bundle is already generated with Rollup. It is, of course, the choice of the maintainers how to play it out, but at least from my perspective, using Babel would add complexity for little benefit. Turf is mostly low-level mathematical code with hot loops. Using syntax extensions would make testing more complex, introduce hard-to-anticipate performance problems, and make debugging error reports harder, because it would be debugging transpiled code and relying on sourcemaps, which rarely work well. |
@tmcw that is a very interesting prospective I could have never thought about NOTE: is this a good site to check to see when ES6 will replace ES5 as the new standard JS? I mean what't the best way to learn when that will become official? |
That's a pretty good resource. Really the good news is that a lot of ES6 features are supported well enough that something like Babel isn't required. My day job (⦿) writes ES6+ code and deploys it without a transpilation step - stuff like classes, let/const, async, generators, and so on all work in the last few major browser versions as well as node. |
@stebogit there is also https://node.green/ that tracks what ES features are supported by the various Node releases |
Hi guys, Love turf. Fantastic library. Many thanks for your efforts! I stumbled across this thread trying to debug an issue using Instead I need to use Anyway, as a developer, I only really use turf in node.js data processing scripts (a little bit in my node/express back end, and not at all in my React front end). We use babel for our back end code so that we can use ES6+, import/export, etc. But with the scripts, it's really nice not having to transpile anything. Just thought I'd add that feedback in light of your current decision-making |
Hi, just my two cents. I've been playing with JavaScript since v1.1, not kidding, Netscape times, you know! And I'm not a foreigner in ES6+ country. But I'm definitely not an architect, I just use the code as a tool to solve a problem, geospatial problem in this very case. So, I'd love to contribute with new modules, but I'm not an expert about the best way to mash-up and serve all the code involved here. I'm not here to give my opinion about the answer (which is 42 regardless the question), I just want to share my experience as a keen contributor and TURF lover, maybe more related to spatial-data-science than coding. I contributed to TURF with several modules time ago, years I mean, some of them made their way into releases, some others were just deprecated with better versions or were just crappy. By those times, nodejs was alien to me, but I was flexible enough to adapt my vanilla code to node modules with just a couple of lines of code. Node is not that hard for this purpose. Now that I'm somehow back, I feel like a noob. Yesterday, I spent the morning porting a classic ESRI's GIS function to TURFJ. It was quite easy, once I already had it ported to SQL and running in production environment, so it was somehow like SQL -> ES6 translation, plus testing. The code worked, my tests were OK, but... BUT, when I tried to fulfill the TURF Contribution Guide requirements, it was just a nightmare. And I just needed to left it there and ask for help in an issue. Not only the Contribution Guide is unfinished and obsolete in some points, but also the mix of JavaScript, TypeScript, requires, imports,... I tried to follow the existing modules as guidance, but I couldn't find two of them following the same rules. So, my 2-3 h project, turned an 8h one, with not even a PR at the end because I couldn't run even the linter or the tests. So, my two cents, finally: I'd go with the simplest way of all of them, the one that doesn't make potential contributors or maintainers run away when they dig in the gears. The one that doesn't make adding a new module a labyrinthic task. And, we're in 2018, a 500kb load in scripts is typically present in any website! I must say, as a final note, that the boilerplate for new modules is such a great help! |
Hey @AbelVM (& related to @danloiterton comments as well) Yeah unfortunately the documentation for contributing to Turf, as well as doco for using Turf, both struggled to stay up to date at the beginning of the year when a bunch of changes happened and they've never caught up. Once the codebase is back in a more stable state I'll revisit them both. |
I just wanted to highlight that the architecture doesn't only impact performance and package sizes. It might also discourage potential contributors if the process to add a new module is like untangling the Xmas lights. |
Just wanted to say I ran into too, in node. |
Are any of you at FOSS4G in Dar es Salaam at the end of the month? Turf development could be a good BOF topic. |
Afraid not @dbauszus-glx - I'm in Australia so I'll likely present on it at FOSS4G Oceania later in the year |
I'm 👎 for removing TypeScript.; we can contribute dev time to help keep it up to date if necessary. Failing that, if you're locked into removing TypeScript, please take control of your own typings. (We can help with that, too.) If you leave it up to DefinitelyTyped, it's all too easy to end up with people making breaking changes in point releases. |
Hi @garthk Who is we? Unfortunantly you're chiming in on an issue several months old and ive already done the bulk of the work to refactor to es6 modules - feel free to check out the v7 branch. And i'll take whatever help is on offer with maimtainig the typings because i dont know how they are supposed to work :) |
Anditi. We've got a few devs working against Turf. At least two of us are happy to edit typings and do some heavy lifting on refactoring to the v7 structure to get the project back on track. We also have a GIS greybeard to help make calls on accuracy issues like #1470. Speaking of, the MAJOR bump to 7 will let you fix #1470, which we'd prefer to having to maintain a fork of 4.7.3 for the rest of our lives. |
Hey @garthk Looks like a neat company (even Aus-based!) - glad to see people putting TurfJS to good use. And it would be great if your company was able to provide a bit of assistance in maintaining TurfJS given your commercial interest. Yep v7 is indeed a major bump on purpose to let us introduce some major changes, amongst which I'm including some major fixes to buffer. The buffer module is also our final dependency on And some input from GIS greybeards would be awesome - unfortunately a few decisions have been made in the past which have had unintended consequences so it would be great to lean on some advice in that space. Would be good to followup via email just to say hello and fill you in a bit more about the current project context, I'll try and get an email to you in the next few days. Cheers |
No worries; use I don't suppose there's any appropriately licensed C++ out there you can run through emscripten to replace JSTS? |
Thankfully after a few years hiatus Turf is back on track with a team of contributors helping on many fronts 🎉 |
The goal for Turf v3 was to create a system in which Turf could be coordinated as a set of modules, operating independently but uniform in style, management, and properly versioned together. Unfortunately, that's no longer the case: the current codebase is heterogenous, hard to contribute to, and the guidelines for releases are nonexistent.
About half of the modules are in TypeScript, and about half of those modules have a tsconfig.json file. About 40% of modules name their
main
exportmain
, and the rest call itindex
. Same with providing amodule
entry point, it's about 50%. 3 modules have a.mjs
module file, the rest have.es.js
.Basically, there are no real homogenous properties of Turf anymore. It's a worst-case scenario for a mono-repo: it makes Turf maintenance feel much more like 'maintaining 55 modules' than it does 'maintaining a mono-repo'.
Turf modules summary
Organizationally, Turf currently has a small chunk of cash, a much much smaller number of long-term maintainers, and zero institutional/company code contributors. This is pretty common for open source projects.
To propose a few ways forward:
Remove TypeScript
The TypeScript transition, while a very nice contribution, is unfinished and offers little to the project. We didn't have many bugs caused by type mismatches before, and the resulting addition of complexity outweighs the benefit.
Would this be unpopular? Probably! But it'd be better to re-achieve stability and releasability, and make the project fit its actual contributors, than to perpetually be in a half-deployable state.
If this is the option, we could do it by basically configuring tsc to generate ES modules.
Move lerna.js back to fixed/locked mode
Per #1247 the lerna config was switched to independent versioning. This makes collaboration harder: the intent of lerna was and is to make versioning automatic so that the project can be released always with
lerna publish
. There should not be a thought process for which modules need updating and releases.Move releases/ back into changelog
This releases directory is the same as a central changelog. Let's move these back to the changelog.
Use standard commits
Again, a common collaboration point - this will automate changelog generation and version more accurately
Write these contribution tips into CONTRIBUTING.md
And probably create a strong process for changing them. This stuff shouldn't become wobbly; Turf should stay in a deployable, simple state that all the maintainers have a grasp on.
(Extreme solution) just move to a singular ES6 module
This would probably be even less popular than removing TypeScript: create one
turf
module (again), now that tree-shaking means that people using Rollup/Webpack/Parcel can efficiently import a subsection of a module without including all of the code. This would drastically reduce complexity and let advanced users continue to get the performance advantages of the current mini-modules, but it would leave people using old-fashionedrequire
etc out in the cold.Honestly I'm also in favor of this extreme solution: it'd be a huge decrease in complexity, and it'd do this for existing users:
This is intended to be a bit of a jumping-off point, none of these ways to victory are quick-fixes and we'd want to attain some consensus before embarking on them. Dividing this work up would also be useful, given that there are few maintainers, or if there is an institutional/company contributor who wants to take on some of this, it'd be more than welcome - Turf is definitely generating a lot of value out there in the world, much more than it captures.
The text was updated successfully, but these errors were encountered: