Standalone Sketch File Format v1.0.0 ☂️ #1
Comments
Impact on SketchThe most important aspect to point out is that using a file spect does not change how Sketch or ScenariosOpening and saving Sketch files typically falls into one of the following scenarios:
Breaking changes, e.g. something like the transition from We'd maintain the It is up to
Conforming to the file format spec1. StepAdding tests to 2. StepThis could potentially (partially) replace the Source formatWe got JSON Schema, Typescript and potentially Swift to choose from.
The decision on using JSON Schema or TypeScript (or Swift) as the source of truth feels like it depends mostly on who's making the changes and what they feel comfortable with. It would be great to get input from the Mac team on this. |
|
Sketch fileformat itself had already multiple transformations. We had long time ago JSON (one big json file), but was not optimal as it's not very efficient. Harder to read over a stream, partly eg. We had a macOS fileformat which was secretly a package/folder. However, lot of sync issues over Dropbox eg. Data/artboards missing. We had a SQLite database, where we even stored blob's. We had quite some corrupted files due parallel operations, saving/syncing the DB eg (more about that in the backlog) The current file format is a ZIP, with multiple JSONs and images saved separately.
I'm all in for a new format, especially to make it more readable and understandable by people, other apps, plugins, tools, but this is quite a big call. What are the exact issues we have currently? The fact that there is no definition of our file format? That it's not generic enough and not using standard/atomic variables but custom classes? I'm also thinking about the collaboration feature. Could that be something that can be done at the format level, versus the tree? 🤔 |
|
@jelledelaender I think there's a misunderstanding here; we're not proposing a new format, instead we're suggesting to introduce a spec for the current format so that different parts, developed by different teams using potentially different languages (currently Obj-C, Swift, JavaScript, TypeScript, Elixir, Go) can be safely assumed to be compatible. For the collaboration, that's a good question and possibly goes beyond the scope of this issue. However, technically speaking a change ripples down to the persistence layer and the way we've been using the spec/TypeScript types allow to work with any sub-tree or specific nodes. |
@jelledelaender ☝️Yep, this. We're not proposing any changes to the existing file format, or how its persisted to disk, rather proposing to formalise the definition somewhere in code. If that definition is in code, it means it can be transformed to be maximally useful in various contexts (TypeScript for frontends, GraphQL schema for backends, maybe in time auto-generate Swift serialisers, and more). And ideally we'd be able to commit to some level of stability in that definition, in such a way that it at least can't be unexpectedly changed with every Sketch release. It might be worth mentioning explicitly that this isn't about recreating the native Sketch model elsewhere, that can and should stay encapsulated within Sketch - this is about defining in code, with a high degree of strictness, what valid Sketch document JSON looks like on disk, or over the wire. This enables type safety in internal and 3rd party app code that might work with Sketch documents, or fragments of Sketch documents, and I think delivers on the vision of an open file format that can act as an interchange format between products. It was actually your comments on #proj-linting channel that really got me thinking about this - when you said we'd have to slave releases of a linting tool for every variant of Sketch releases https://bohemiancoding.slack.com/archives/CK9MW0PL2/p1563203566003800 That imposes a very large burden on an internal linting tool, but one that could just about bearable. But in order to deliver the sort of linting experience we want to, where 3rd parties are able to craft custom rules for their diverse internal workflows and standards, to have the caveat that they have to build rules against an undocumented and unspecified file format that could change under their feet with every new Sketch release ... well, it's basically a non-starter IMO. And it's not as if linting is unique in this regard - any product that wants to work with Sketch documents is currently built on shakey foundations without a defined file format. And when additional stability (or at least the illusion thereof) is required by 3rd parties they have to recreate the format themselves on a case by case basis, and attempt to manually keep in sync with the sorts of files Sketch outputs. E.g. here's an example in Amazon's Sketch Constructor tool https://github.com/amzn/sketch-constructor/tree/master/models So instead of everyone having to create their own file format spec, this proposal is also about us doing that ourselves properly, and also own the versioning and change management in a way only we're able to.
I don't have many opinions about what can and should be improved within the file format. But I do think that properly defining it could make future changes less painful. |
|
Some ramblings from the Mac side... I have no experience of TypeScript and don't really understand how it would be used to define the format, but from what I'm reading here it sounds like a sensible choice, especially given that we can generate a JSON schema directly from it. Regarding code generation: We currently use the Sketch.xcdatamodel alongside a custom tool (Coma) to generate boilerplate source code. I don't think that the technology we use to define the format's source of truth necessary has to be the same as the source we generate this code from, but the .xcdatamodel is creeking at the seams a little and if we can kill two birds with one stone that would be useful. With that in mind I have some questions: Firstly - while Swift is the new shiney, I'm not convinced we want to change the model to be Swift any time soon. For one thing there's a posibility that we'll develop a new rendering engine in C++ and I think the interopability between C++ and Obj-C would better than between C++ and Swift. So is there a mechanism to generate Obj-C code from TypeScript? I suspect the answer is yes in that we could generate JSON and use Mustache to generate the Obj-C code as we do at the moment. Secondly - does TypeScript offer a way to extend the model without editing. This is a limitation with the Sketch.xcdatamodel. A recent example of this, it would have been nice to supply a separate config file to define attributes required for plugin diffing rather than adding them to the .xcdatamodel. Also, what additional tools are needed to compile/run TypeScript. Currently Coma is included in the Sketch repo meaning that a new developer can more or less just clone the repo and compile in Xcode to be up and running. A concern with having to install third party tools for this is that they could break going forward - e.g. they don't run on 10.16 when it's released. Change management:
That scares me. While I can completely see the logic I worry that we'll lose a lot of the agility that gives us an edge over our competitors. I would hate to see Figma releasing features before us just because we were waiting for a point where we're allowed to change the file format. Just to follow on from what @christianklotz said above, it might be useful to describe how we handle rolling changes on the Mac side: Forward migrations (opening older documents) are the simplest to handle; when we change the version number we can add additional code telling Sketch how to migrate old objects/properties to the new format. When Sketch opens a file with an older version number it automatically runs this migration code to bring the document up-to-date. Opening newer documents in older version of Sketch is trickier, but with care we can manage this too. A good example of this is the recent #25151 where we only write values if they don't match the default. Obviously, if we just stop writing these values in v58, v57 wouldn't be able to read v58 files so the way we deal with this is as follows: There are 2 version numbers stored in a Sketch document; a current version and a compatibility version (these are defined in MSDocumentVersion.h - it seems likely that this file is something we should consider integrating into all of this work). As of v57 the version is 119 and the compatibility version is 99, which means v57 will continue opening documents as long as the compatibily version is <= 119. At v58 we will add the ability to read files where the defaults are missing, but we won't yet omit writing them. We will also increase the current version number to 120. The compatibility version will stay at 99 meaning that v57 can still open v58 files. At some later point we'll stop writing the default values. At this point we'll change the compatibility version to 120. Now, when v57 tries to open the document it'll read the compatibility version and as that (120) is greater than the current version (119) it will fail, presenting the user with an error. However, it the user tries to open the document in v58, the compatibility version will match the current version and we can open the file. Further, the user could then save the file from v58 and this would be openable by v57. A different example would be symbol overrides. Originally these were stored as a dictionary before later being moved to objects in their own right. To allow old versions of Sketch to understand the newer files we had a period of overlap where both the old format dictionary and the new override objects were writen. I think it would be good in situations like this to be able to publish a format containing both the dictionary and the objects, but to be able to mark the dictionary as deprecated. |
The useful property of TypeScript is that it has a well defined and documented AST, so a TypeScript file is easily transformed into a pure data representation. So then it's a case of writing shell scripts (in Node) that ingest that data, and output it in a new format, for example JSON Schema. Here's an example from an existing tool - TypeScript on the left, and the generated Obj-C on the right.
Yep, you're correct - ultimately the file format would end up as plain JSON, and the tooling you use to generate code is up for grabs.
TypeScript is a Node tool, so Node and npm are required. I don't think breaking is too much of a concern, it's built by Microsoft and currently sees about 6.5M weekly downloads, mostly by web developers who all use Macs. That said, I'm not sure where we'd want to store the file format source of truth itself ... on its own somewhere, or bundled into the Sketch repo? If it was on its own then the toolchain used to build it wouldn't be a direct dependency of Sketch, since I guess Sketch would consume a plain JSON representation. Do you have any opinions about whether the source of truth is in Sketch, or not? All that TypeScript chat aside though, following a sprint planning discussion with @christianklotz this morning, we've more or less decided to have a stab at writing a basic/MVPish version as plain JSON Schema. Got to start somewhere, and writing the format in JSON seemed like a simple place to begin. Should be useful to get the ball rolling at the very least.
This an interesting one, because I think both TypeScript (and JSON Schema) are more concerned about describing the shapes of objects, rather than recreating classically OO inheritance patterns. So I think with both approaches extending is OK, but you run into problems with attempting to override existing properties with new types using their own semantics. That said, JSON Schema is just JSON, so it's easy enough to write scripts that manipulate it as pure data. I'm sure we could come up with a nice way to "patch" a file format, temporarily or otherwise, with extra props. The thing with JSON is that there's really quite a mature ecosystem of specs around now, case in point - http://jsonpatch.com/
Yep, a few others raised concerns about this. This is about the tension between Sketch as a consumer product that requires frequent improvements, and Sketch as a platform/format that needs to co-exist within toolchains. I suppose the best thing that can be said is that by formalising and publishing a file format spec, we don't overtly harm the former interest, and actively improve the latter.
Since the current objective of the file format schema is just to describe how a Sketch file should look on disk, it doesn't seem like it would initially get in the way of how you currently handle rolling changes. The schema would just define the current and compat version properties as numbers, say where they appear in the JSON, maybe add some descriptions etc. An interesting question to me is that, hypothetically speaking, if a Sketch file "just" declared what version of the file format it was saved as (e.g. file format
For sure, we could mark properties as deprecated in JSON Schema. At that point it's just a boolean flag, and we use it to adjust what may get generated, whether a deprecated label on some website documentation, or affecting code generation somehow. |
|
Once we got something shareable, we already got some people who've expressed interest in giving it a spin.
|
|
We've now release v1 of the file format https://www.npmjs.com/package/@sketch-hq/sketch-file-format, so closing this issue since it feels like we've achieved that initial milestone. |
Oh boy this would have been great to know! 😄 |
Progress
This issues discusses how a v1.0.0 of a standalone Sketch File Format project might work and look like.
A standalone Sketch File Format would have the following important properties:
✂️ Hand-crafted vs auto-generation
We've already tried auto-generating a file format specification from the Sketch code base. It was an incredibly useful learning exercise, but has critical drawbacks:
Therefore, maintaining a hand-crafted file format as a product seems like the best next step to explore.
👑 Source format
The source format defines the technology we use to maintain the file format source, but isn't prescriptive about the formats that can be derived from it. Indeed, the objective here is to pick a source format that enables fluent transformation to as many other output formats as possible.
There are two main candidates:
JSON Schema
JSON Schema itself is an obvious choice. It has rich semantics which allows it to fully describe complex models, and has been widely adopted with tooling support in many languages and platforms. The JSON Schema specification process has been adopted by the IETF and sees active development.
A potential downside is that JSON Schema is just raw JSON, and as such the experience of maintaining a complex/large JSON Schema document by hand could have some challenges. We could look into JSON Schema $refs to split the schema into manageable chunks.
TypeScript
Defining the file format in code using TypeScript would bring a few benefits:
The TypeScript AST is well documented, and would serve as a robust data format to enable transformation to other formats. We could write our own TypeScript to JSON Schema transformer for example, or use existing tools like typescript-json-schema. Tooling for TypeScript to Swift transformation exists in the wild too, e.g. quicktype.
We'd have to ensure that we can sufficiently describe the existing file format model in TypeScript before choosing this route, since the range of type information and their interrelationships it can describe is more limited than JSON Schema (this could turn out to be a good thing - since its also simpler).
Swift
There could be a compelling case made for using Swift for the source format, since the most important consumer of the file format will be Sketch.app. The feasibility of this would hinge around the suitability of using the Swift AST as a source data format for onward transformation into JSON Schema, TypeScript and others.
Is the Swift AST sufficiently mature? What does the existing tooling and documentation around it look like? Does it limit the potential for 3rd parties to easily write their own transformation scripts? This would need investigating by a native developer.
💁 Output formats
We would expect the community to use the file format to generate diverse formats that work with their specific workflows and toolchains, therefore we don't need to be exhaustive in the output formats we initially implement.
A v1.0.0 of the file format could simply include the source format itself, whatever it is. Additional output formats that we're certain to need internally (e.g. TypeScript, or Swift) could be looked at as a follow on task.
Rather than pollute the core file format repo with generator code for other formats, these could be housed in separate repos (GitHub Actions could eventually be used to trigger builds when a new file format is released).
🌳 Structure
The following structural elements and types should be describable by the file format,
layersarray can only contain elements of union typeLayer, for example)_class, orcolorProfileproperties etc)There's no beating around the bush - the main driver of change in the file format would be Sketch.app.
Rather than deny that reality the objective is to formalise the relationship, ensure changes to the file format are explicit and deliberate and to implement a process where new versions of the file format are released ahead of time, or concurrently with the version of Sketch.app that supports it.
The change management process could look something like,
🤖 Hosting
The primary hosting mechanism could be tagged GitHub releases on the file format repo.
Thereafter, derived output formats could be hosted in whatever fashion makes most sense for their community/ecosystem. E.g. npm for TypeScript types etc.
🤔 For further discussion
How will the file format fit into the native Sketch.app workflow?
The responsibility here will lie with the Sketch dev team to figure out an approach for ensuring the versions of Sketch they release output documents that validate against the file format.
It's important to note that, technically speaking, a v1.0.0 of a standalone file format should precisely match the schema of files outputted by the current version of Sketch. So initially the changes to the Sketch development process might centre more around testing, than large internal changes.
We need input from native developers but a few strategies could include,
The text was updated successfully, but these errors were encountered: