Multiple conditional / mode values for a single design token

[lukasoppermann]

Hey, I am running into an issue with different scenarios where I need multiple values for on token.

In all cases the value I want is tied to a kind of mode and will be returned within one file, but within selectors or media queries.

Examples

Example 1: dark & light mode

:root,
[data-color-mode="light"] {
  --column-bgColor-overlay: #ec51544d;
}

[data-color-mode="dark"] {
  --column-bgColor-overlay: #ec51544d;
}

Example 2: responsive tokens

:root {
  --spacing-margin: var(--base-size-16);
}

@media (min-width: 768px) {
  :root {
    --spacing-margin: var(--base-size-40);
  }
}

@media (min-width: 1280px) {
  :root {
    --spacing-margin: var(--base-size-24);
  }
}

Problem

Previously I have used multiple files, e.g. colors.light.json and colors.dark.json. However now, I have component tokens in here as well. Those tokens are defined within the component folders as individual files. E.g. components/table/table.json.

Defining files for both modes and media queries for each component does not make for a good workflow. This is why I'd like to define them in one file. However, since the tokens must have the same name, the only approach I have found is the single token method described by @dbanksdesign. Since it has the huge downside of not running transformers on anything but the value property, it feels like a bad solution.

I am wondering if there is any other way of getting this to work.

I was thinking of something like:

{
  token: {
    value: [
      {
         value: "{base.size.16}" // default
      },
      {
         value: "{base.size.40}"
         $extensions: {
           "myOrg.mediaQuery": "@media (min-width: 768px)"
         }
      },
      {
         value: "{base.size.24}"
         $extensions: {
           "myOrg.mediaQuery": "@media (min-width: 1280px)"
         }
      },
    ]
  }
}

However, this needs to run through the transformers, which does not work. If I would get those as transformed values in my formatter, it would be all great.

[jorenbroekema]

design-tokens/community-group#210 this is an interesting read on theming, it's probably the #1 topic that we need to find a consensus on, how to approach theming. Until then it's a bit difficult to implement a theming approach into style-dictionary itself, and right now it's kind of up to the user to use Style-Dictionary in creative ways to get the themed outputs as they like.

My personal opinion on this is that theming is pretty broad, can get quite complex and therefore must be scalable. It's not just about light/dark mode anymore, and therefore I quite like the approach we take at Tokens Studio where tokensets (separate design token json files) are loosely coupled to themes and theme-specific overrides tend to live in separate tokensets from the base tokens, acting as overrides. Imagine having just a light and dark mode, then you have 2 permutations, light and dark, but if you introduce multiple dimensions of theming this can go up exponentially. The way we approach that right now is that a separate StyleDictionary instance is created for each permutation of theming, so if you got 2 modes and 3 viewports that would be 6 output files at minimum (3*2 theme permutations). In your frontend app or design system, you will use some kind of utility to load the correct output file (e.g. CSS stylesheet) based on the user's theme selection, in this way you never load redundant styles. This approach is demonstrated in lion-example (demo) which I will probably built upon a bit more in the future, let me know if you have questions about it!

The problem with combining it into a single token and also into a single output file e.g.:

:root,
[data-color-mode="light"] {
  --column-bgColor-overlay: #ec51544d;
}

[data-color-mode="dark"] {
  --column-bgColor-overlay: #ec51544d;
}

is that you're always loading a ton of unused styles because you are usually not in light and dark mode simultaneously.

Your multiple-file approach is therefore what I would personally recommend you switch back to, but you'll likely need 3 layers:

• primitives, not dependent on themes
• semantics, it is this layer that changes based on themes
• component tokens, consumes from semantic tokens, but are themselves not directly theme-specific, so you don't have a button-light.json, just a button.json which consumes refs from light.json or dark.json depending on the theme selection.

This extra theme-specific semantic layer should allow you to not have to create component token files for each theme permutation, because it's handled on the semantic level.

That said, you're free of course to take the single file approach, putting theme permutations inside individual tokens and combining it in a single output file, but imo this is much less performant later down the line and less scalable as your themeability grows. And you'll have to create a custom transform that goes through the theme options and groups token values by theme properly.

However, this needs to run through the transformers, which does not work. If I would get those as transformed values in my formatter, it would be all great.

Can you elaborate on this? Inside transforms (and also inside formats) you do have access to $extensions metadata, and if there are references in that metadata that need to be resolved in order for the transform to do its work, you can use this feature to defer transformation until such references are resolved: https://v4.styledictionary.com/reference/hooks/transforms/#defer-transitive-transformation-manually

[lukasoppermann]

Hey @jorenbroekema, thanks for the detailed reply.

Let me first mention, that I am not the maintainer of the repo, it is another part in my org, so I want to keep the approach they know or at least something as close to this as possible.

The first approach with the SD instances is not that great for us, because our files are manually generated. This means we would manually need to create the files and update them in the script that combines them every time we create a new component.

For approach two, the overhead of unused css is no big concern for us. A couple lines more or less have no significant impact on loading time.

My problem is that I don't know how to set up one json file with multiple values for a single token, that still runs transformers over all values.

What I will try now is:

1. Can I use a naming convention:

{
  tokenName: {
    `@#0`: {
       type: "color",
       value: "{base.color.red}",
       mode: "light" // simplified example, I will do something within $extensions
    },
    `@#1`: {
       type: "color",
       value: "{base.color.darkRed}",
       mode: "dark" // simplified example, I will do something within $extensions
    }
  }
}

2. In the formatter replace names
   In the formatter I just run a function that removes @3\d+ from the name.

3. If this works, I could build a parsers, that converts something like:

{
  tokenName: {
    type: "color",
    values: [
    {
       value: "{base.color.red}",
       mode: "light" // simplified example, I will do something within $extensions
    },
     {
       value: "{base.color.darkRed}",
       mode: "dark" // simplified example, I will do something within $extensions
    }
    ],
  }
}

to the output from above. This may be a decent solution for now, if it works. The only issue I see, is that it relies on a formatter to output the correct tokens, which I really don't like.

However, this needs to run through the transformers, which does not work. If I would get those as transformed values in my formatter, it would be all great.

Can you elaborate on this? Inside transforms (and also inside formats) you do have access to $extensions metadata, and if there are references in that metadata that need to be resolved in order for the transform to do its work, you can use this feature to defer transformation until such references are resolved: https://v4.styledictionary.com/reference/hooks/transforms/#defer-transitive-transformation-manually

If I do something like like this:

{
  tokenName: {
    type: "color",
    value: "{base.color.red}",
    valueDark: "#440000",
  }
}

Transformers will NOT transform valueDark, since they only run on value properties.

[luupanu]

I got a similar thing working using a custom formatter, basically something like:

{
  "token": {
    "value": "12px",
    "breakpoints": {
      "mobile": "14px",
      "tablet": "16px"
    }
  }
}

enum Breakpoint {
  tablet = 800,
  mobile = 500
}

const SPACING = '  ';

const indentString = (str: string, indentationLevel: number = 0) => {
  const indentation = SPACING.repeat(indentationLevel);
  return str
    .split('\n')
    .map((line) => indentation + line)
    .join('\n');
};

const formatTokens = (formatter: string, args: FormatterArguments, allTokens: TransformedToken[]) => {
  return StyleDictionary.format[formatter]({
    ...args,
    dictionary: {
      ...args.dictionary,
      allTokens
    }
  }).trim();
};

const formatTokensWithBreakpoints = (
  formatter: string,
  args: FormatterArguments,
  breakpoint: Breakpoint
): string | undefined => {
  const breakpointKey = Breakpoint[breakpoint];

  const tokensWithBreakpoints = args.dictionary.allTokens
    .filter((token: TransformedToken) => token.breakpoints?.[breakpointKey])
    .map((token: TransformedToken) => ({
      ...token,
      original: { ...token.original, value: token.original.breakpoints[breakpointKey] }, // replace token.original.value to make outputReferences work
      value: token.breakpoints[breakpointKey] // replace token.value with the breakpoint
    }));

  if (tokensWithBreakpoints.length === 0) return; // if no breakpoints, return undefined

  const formattedTokens = formatTokens(formatter, args, tokensWithBreakpoints);
  const formattedMediaQuery = `@media (min-width: ${breakpoint}px) {\n${indentString(formattedTokens, 1)}\n}`;

  return formattedMediaQuery;
};

StyleDictionary.registerFormat({
  name: 'css/my-custom-formatter',
  formatter: (args: FormatterArguments) => {
    const defaultFormatter = 'css/variables';

    // Format tokens without breakpoints
    const formattedTokens = formatTokens(defaultFormatter, args, args.dictionary.allTokens);

    // Format tokens with mobile breakpoints
    const formattedMobileTokens = formatTokensWithBreakpoints(defaultFormatter, args, Breakpoint.mobile);

    // Format tokens with tablet breakpoints
    const formattedTabletTokens = formatTokensWithBreakpoints(defaultFormatter, args, Breakpoint.tablet);

    const allFormattedTokens = [formattedTokens, formattedMobileTokens, formattedTabletTokens].filter(Boolean); // filter out undefined / empty strings
    return allFormattedTokens.join('\n') + '\n';
  }
});

I haven't tested it out with transforms or with v4, I think if you're using references in order to make them work you might need to use transitive, deferred transforms as @jorenbroekema said.

Other than that this solution replaces token.value & token.original.value so at least normal transforms should work.

There might be some errors in the code example above.

[lukasoppermann]

Hey @luupanu,

I think (I am not sure), that transforms are run BEFORE the formatter. This means they would not be run on your values. Would be great if you could test it, e.g. if a pxToRem transformer would work on all values.

[jorenbroekema]

@lukasoppermann

The first approach with the SD instances is not that great for us, because our files are manually generated. This means we would manually need to create the files and update them in the script that combines them every time we create a new component.

Can you elaborate on this, I don't think I understand what this means. Which files are manually generated? If you use Style Dictionary, that usually means by definition your files are automatically generated. If you're talking about your tokens files, then that doesn't matter because even though you create a new tokens file for every new component, because they all consume from the semantic layer (which is your themes-specific layer) which you only have to create once, this doesn't matter?

For approach two, the overhead of unused css is no big concern for us. A couple lines more or less have no significant impact on loading time.

I don't mean to overstep here but I've heard people say this a lot and I'm a bit cynical about this viewpoint. Unused CSS is almost always a significant performance concern, especially for design systems.
Let me throw some numbers your way to convince you. Imagine a pretty basic design system with light and dark mode, as well as 3 breakpoints, this means you have 2 * 3 = 6 possible theme permutations. Let's imagine this design system has 20 components, each component on average consumes 20 design tokens that are theme-specific, which is a very conservative estimate.
In terms of theme-specific CSS we're talking 6 * 20 * 20 = 2400 lines of CSS, 1/6th of which is relevant which means 2000 lines of unused CSS, and that's only the components. If we add to this that application and feature teams will usually also import the semantic layer itself to structure and style their app/feature, you could estimate that the amount becomes 1.5 as much so let's say 3000 lines of unused CSS. Depending on your token names and values and whether you outputReferences, 1 line of CSS may be somewhere between 10-50 characters so let's go with 25. 25 characters = 50 bytes. Multiple that by 3000 and you've got 1.5 megabytes of unused CSS. I've also tried this with super conservative estimates resulting in at least 7.5 kb, but even 7.5 kb is very significant for users with poor connections and can add whole seconds to your load times. Let alone what 1.5 megabytes does, which could be way way higher if you've got a more complex design system, and every theme permutation makes this exponentially larger, and keep in mind that the amount of theme permutations has an exponential relationship with the amount of theme dimensions, we've only used 2 in my example.

Obviously I've made a lot of assumptions here but it's hard to believe for me that combining all theme results into a single CSS file isn't a problem..

Transformers will NOT transform valueDark, since they only run on value properties.

Okay I see, true, that's not something that will change or that's easy to work around.

With regards to your suggested solutions, putting everything inside the value prop means that your token values are always arrays or objects which is super inconvenient to work with from the perspective of value transforms, as you pointed out using valueDark or something similar means they aren't transformed at all, so that leaves you with the token group naming convention which could work if you do something like this in your formatter:

// assuming this convention
const tokens = {
  tokenName: {
    `@#0`: {
       type: "color",
       value: "{base.color.red}",
       themes: {
         mode: "light" // simplified example, I will do something within $extensions,
       }
    },
    `@#1`: {
       type: "color",
       value: "{base.color.darkRed}",
       themes: {
         mode: "dark" // simplified example, I will do something within $extensions
       }
    }
  }
}

const lightTokens = [];
const darkTokens = [];

dictionary.allTokens.forEach(token => {
  const reg = /@#\d+$/g;
  if (token.name.match(reg ) {
    token.name = token.name.replace(reg , '');
    if (token.mode === 'light') {
      lightTokens.push(token);
    } else if (token.mode === 'dark') {
      darkTokens.push(token);
    }
  }
});

// now that your tokens are properly grouped and named, you can output them inside CSS selectors based on group

My recommendation is still using token overrides + multiple SD instances and splitting outputs, not only because of the perf concerns but also because it's compliant with the DTCG spec and requires no style-dictionary "hacks" or otherwise brittle conventions whatsoever, and it's also very scalable. I'm happy to hop on a call sometime and see if I can get a better understanding of your use case and what would be needed to go for that approach

[luupanu]

Hey @luupanu,

I think (I am not sure), that transforms are run BEFORE the formatter. This means they would not be run on your values. Would be great if you could test it, e.g. if a pxToRem transformer would work on all values.

You're correct, transforms are run before the formatter. I guess one could manually apply the transforms before applying the formats in by declaring them in some global variable and applying them using the exposed functions like transformObject in style-dictionary but that's going to be super hacky for sure, and from what I've understood v4 removes the exposed private functions anyway.

Not sure why there is a args.platform.transforms in FormatterArguments that only exposes type and transitive from the transform objects? Could there also be a potential change where one would get the actual transforms instead there?

[lukasoppermann]

I don't mean to overstep here but I've heard people say this a lot and I'm a bit cynical about this viewpoint. Unused CSS is almost always a significant performance concern, especially for design systems.
Let me throw some numbers your way to convince you. Imagine a pretty basic design system with light and dark mode, as well as 3 breakpoints, this means you have 2 * 3 = 6 possible theme permutations. Let's imagine this design system has 20 components, each component on average consumes 20 design tokens that are theme-specific, which is a very conservative estimate.
In terms of theme-specific CSS we're talking 6 * 20 * 20 = 2400 lines of CSS, 1/6th of which is relevant which means 2000 lines of unused CSS, and that's only the components. If we add to this that application and feature teams will usually also import the semantic layer itself to structure and style their app/feature, you could estimate that the amount becomes 1.5 as much so let's say 3000 lines of unused CSS. Depending on your token names and values and whether you outputReferences, 1 line of CSS may be somewhere between 10-50 characters so let's go with 25. 25 characters = 50 bytes. Multiple that by 3000 and you've got 1.5 megabytes of unused CSS. I've also tried this with super conservative estimates resulting in at least 7.5 kb, but even 7.5 kb is very significant for users with poor connections and can add whole seconds to your load times. Let alone what 1.5 megabytes does, which could be way way higher if you've got a more complex design system, and every theme permutation makes this exponentially larger, and keep in mind that the amount of theme permutations has an exponential relationship with the amount of theme dimensions, we've only used 2 in my example.

Obviously I've made a lot of assumptions here but it's hard to believe for me that combining all theme results into a single CSS file isn't a problem..

So first of all I think we can drop this discussion, as it is not my place to change this in the project where I am helping out.
Secondly it is relatively small, so the global file has a gzipped difference of 158 byte. Most component files only have one or two lines that need to be specifically overriden, so they actual have an overall diff of 75 byte or similar.
Lastly, I could split them into files per mode with style dictionary and still run into the issue I experience. So even if this would be taken on, I would want to make this work.

The first approach with the SD instances is not that great for us, because our files are manually generated. This means we would manually need to create the files and update them in the script that combines them every time we create a new component.

Can you elaborate on this, I don't think I understand what this means. Which files are manually generated? If you use Style Dictionary, that usually means by definition your files are automatically generated. If you're talking about your tokens files, then that doesn't matter because even though you create a new tokens file for every new component, because they all consume from the semantic layer (which is your themes-specific layer) which you only have to create once, this doesn't matter?

Sure.

1. We build our files by hand
2. We have ~ 30 components, not all, but a good amount have custom tokens
3. Each component has individual json files

If we wanted to create files for each mode, we would have to do something like this:

• card/color.light.json
• card/color.dark.json
• card/size.sm.json (this is a view port / media query, I just use a t-shirt size for simplification)
• card/size.md.json
• card/size.lg.json

Doing this 15 or 20 times is very annoying. I also need to load all those in my script and make sure the output file name is okay. So either I have to do something like this below and loop over the array. In this case I have to add to this array whenever a new component or file is added to the component.

[{
  output: 'card.css',
  files: [
    // files from anove
  ]
},
// next component
]

or I just use a glob like /components/**/*.json and have to establish a naming convention that the folder name is somehow use to generate the output file, etc.

Maybe I am missing an option here? If so, please let me know.

Also @jorenbroekema thanks for the formatter, this is what I was thinking of exactly.

[jorenbroekema]

@lukasoppermann after our call I drafted something to try and get the format you want while keeping the token authoring process somewhat easy for the users. Here's a link to the configurator to showcase it

It was a little bit harder to get working than I was expecting initially.

It expects tokens to be structured something like this:

{
  "colors": {
    "foreground": {
      "type": "color",
      "value": "<themed>",
      "$extensions": {
        "com.bar.foo": {
          "themes": {
            "mode": {
              "light": "#ff0000",
              "dark": "#cc0000"
            }
          }
        }
      }
    }
  }
}

which is then converted to

{
  "colors": {
    "foreground": {
      "@#0": {
        "type": "color",
        "value": "#ff0000",
        "mode": "light",
        "$extensions": {
          "com.bar.foo": {
            "themes": {
              "mode": {
                "light": "#ff0000",
                "dark": "#cc0000"
              }
            }
          }
        }
      },
      "@#1": {
        "type": "color",
        "value": "#cc0000",
        "mode": "dark",
        "$extensions": {
          "com.bar.foo": {
            "themes": {
              "mode": {
                "light": "#ff0000",
                "dark": "#cc0000"
              }
            }
          }
        }
      }
    }
  }
}

One really important caveat here is that you cannot have any tokens referencing these themed tokens.
This is because we alter the token group structure for these tokens and add these special @#0 to the token path, and the token path is used to resolve references.
Conceptually this caveat also makes sense; it isn't just a technical challenge. After all, if we reference a token that has multiple possible values depending on the theme, then how are we to resolve the reference to the resolved value without knowing the current theme?

Hopefully this helps, obviously adjust the code to your needs.

And as goes without saying, I definitely do not recommend this over the other theming approach that I suggested, it's a pretty hacky solution and atm I don't really see a more elegant way to combine multiple theme values into a single token in a way that's scalable.

[jorenbroekema]

@luupanu thanks for sharing your approach btw

The main issue with this token format:

{
  "token": {
    "value": "12px",
    "breakpoints": {
      "mobile": "14px",
      "tablet": "16px"
    }
  }
}

Is that if you want to run value transforms on this, it will only apply to "value" (12px) and never on the mobile (14px) and tablet (16px)

I guess one could manually apply the transforms before applying the formats in by declaring them in some global variable and applying them using the exposed functions like transformObject in style-dictionary but that's going to be super hacky for sure, and from what I've understood v4 removes the exposed private functions anyway.

Yes correct, definitely very hacky and not something we will support in v4

Not sure why there is a args.platform.transforms in FormatterArguments that only exposes type and transitive from the transform objects? Could there also be a potential change where one would get the actual transforms instead there?

Hm I think that should only give you the names of the applied transforms and nothing else?
What you want is options.transform (and options.transformGroup if the platform uses transformGroup) if you want to know about which transforms are registered on the SD instance, and then filter those by the platform.transforms/platform.transformGroup to see which of those are actually applied...

It's definitely a bit tricky and all this is quite poorly named as well e.g. options.transform being singular form rather than plural. This is something I'm going to improve on significantly as the last breaking change that will make v4: #1049

[jorenbroekema]

Okay seems this doesn't work in the way I was personally expecting, but I'll raise an issue for this and probably refactor this a bit to make platform.transforms in formatter args just be the array of strings (transform names), because this is a bit confusing for users if they want to make use of the platform config inside a formatter