TC39 expertise wanted for "protected state" in web classes

[domenic]

Hi TC39, apologies if this is not the right venue, but I wasn't sure if there was a better one.

For web components APIs, we need to define something very similar to classical "protected state". Concretely, we want to give abilities to manipulate HTMLElement's internal state to anyone who derives from HTMLElement, while hiding it from arbitrary consumers of HTMLElement instances.

The original thread is at WICG/webcomponents#758. We found several potential patterns, listed at WICG/webcomponents#758 (comment). The one we've tentatively settled on is the following:

// This would be implemented by the browser, but shown here in JS for readability
class HTMLElement {
  #protectedState = 0;
  #internals = {
    set: (x) => this.#protectedState = x,
    get: () => this.#protectedState
  };
  
  // Can only be called once, and only if the `needsInternals` static is truthy.
  getInternals() {
    if (this.constructor.needsInternals && this.#internals) {
      const internals = this.#internals;
      this.#internals = null;
      return internals;
    }
    throw new TypeError("Invalid attempt to get internals");
  }
}

// This is the subclass
class CustomElement extends HTMLElement { 
  static needsInternals = true;

  #internals = this.getInternals();

  // Public methods can use this.#internals as appropriate
  increment() {
    this.#internals.set(this.#internals.get() + 1);
  }
}

const ce = new CustomElement();
ce.increment(); // will manipulate the private state

ce.getInternals(); // will throw; protected state does not leak to consumers

Notice the needsInternals check in particular. The idea is that any correct subclass that wants to use the internals will set needsInternals to true, and then call this.getInternals() in the constructor.

Thus, the only way the internals (i.e., the protected state accessors) could leak is if some subclass declares needsInternals = true, but runs arbitrary class-consumer code before it calls this.getInternals(). Because then the class-consumer code could itself call instance.getInternals(), and start messing with the protected state. So, subclasses should not do that; if they do, they have lost encapsulation abilities for their instances.

---

So. That's one pattern. But are there better ones that TC39 would advise? This is the first time the need for such a thing has come up on the web or in the JS spec, but it may set a precedent going forward. We'd love your thoughts before we commit to building these APIs on the web.

[littledan]

Seems like that would work in terms of information flow (EDIT: it's hackable, but it's fine if non-hack-ability isn't a design goal), but it's a little clunky. Maybe it could all get wrapped up in a decorator, that you apply to a private field declaration to magically transform it into the private internals getter?

[domenic]

We're not concerned about the authoring experience for the authors of HTMLElement; that class is authored in C++. We're wondering about the public API for subclass users, the surface area of which is basically the needsInternals + getInternals() pieces.

Or do you see a way to make that nicer using decorators?

[littledan]

Right, I meant it seems a little clunky for custom elements authors, to have to set this needsInternals thing in addition to getInternals. Here's how I think we could make your above code sample look:

// This would be implemented by the browser, but shown here in JS for readability
class HTMLElement {
  #protectedState = 0;
  
  // Decorator for a field declaration
  static internals(desc) {
    assert(desc.kind === "field");
    assert(desc.placement === "own");
    assert(desc.initializer === undefined);
    assert(typeof desc.key === "object");  // PrivateName
    return {
      kind: "method",
      placement: "own",
      key: desc.key,
      descriptor: {
        get() { return this.#protectedState; }
        set(v) { this.#protectedState = v; }
      }
    };
  }
}

// This is the subclass
class CustomElement extends HTMLElement { 
  @HTMLElement.internals #internals;

  // Public methods can use this.#internals as appropriate
  increment() {
    this.#internals = this.#internals + 1;
  }
}

const ce = new CustomElement();
ce.increment(); // will manipulate the private state

Note that you could make @HTMLElement.internals actually decide on its behavior based on the name of the private field declaration it's used on, if that's nice for ergonomic reasons, by using PrivateName.prototype.description. FWIW, I can't think of a method to get at the internal state from outside of the class if this decorator-based approach were adopted without messing with the HTMLElement object itself. (turned out to be wrong, see below.)

[domenic]

Very interesting, thank you! Do you think that's the pattern you (and possibly TC39) would recommend for future web platform and Ecma-262 "protected state" scenarios?

[littledan]

I don't know whether TC39 will recommend anything in particular, but I've been recommending this pattern to folks whenever they ask about protected state in class features (which comes up a lot in the issue tracker, as well as at TC39 meetings). See https://github.com/tc39/proposal-decorators/blob/master/friend.js for some more mini-frameworks for protected-ness that you could play with (there's some vague talk about eventually making a decorator standard library, which presumably could consider adding these), but I don't see any problems with using decorators in a one-off way as in the above code sample.

[domenic]

Some points brought up in IRC about the original solution, transcribing here:

• By doing Function.prototype.needsInternals = true, one can get rid of the slight protection that needsInternals provides to classes that were authored before we introduced this API (i.e. classes that don't call getInternals()). So, needsInternals isn't really protecting much.

• Because getInternals() throws when called twice, subclasses who also want to access the internals are prohibited from doing so, and need to negotiate with their parent class for some way to get access to the internals object. (Perhaps by reimplementing the same protected dance again.) This is not great.

The decorators based approach does not have either of these drawbacks. It just has the drawback of blocking on decorators, which ... well, I'll stop there.

[bakkot]

@littledan

FWIW, I can't think of a method to get at the internal state from outside of the class if this decorator-based approach were adopted without messing with the HTMLElement object itself.

Can't an adversary just do

HTMLElement
  .internals({ kind: 'field', placement: 'own', initializer: void 0, key: {} })
  .descriptor
  .get
  .call(OBJ)

?

This is hard to avoid. protected doesn't make much sense given JS's dynamic inheritance, at least as a defense against a dedicated adversary.

@domenic

By doing Function.prototype.needsInternals = true

You can solve that by checking having HTMLElement require needsInternals to be an own property of the constructor (or, if you want it to be heritable, by manually walking up the prototype chain checking for needsInternals and aborting if you reach Function.prototype).

[littledan]

Oh, right. This is exactly what Ron was proposing that we try to guard against by making less powerful private keys... But I am not sure exactly how that could work to prevent this.

[bakkot]

One other possible pattern:

class HTMLElement {
  #protectedState = 0;
  #internals = {
    set: (x) => this.#protectedState = x,
    get: () => this.#protectedState
  };

  constructor({ internals } = {}) {
    if (internals != null) {
      internals.get = #internals.get;
      internals.set = #internals.set;
    }
  }
}

// This is the subclass
class CustomElement extends HTMLElement { 
  #internals;
  constructor() {
    let internals = {};
    super({ internals });
    this.#internals = internals;
  }

  // Public methods can use this.#internals as appropriate
  increment() {
    this.#internals.set(this.#internals.get() + 1);
  }
}

(There are other possible designs along these lines, like passing an init callback which will get called with internals, rather than passing an internals object which gets mutated.)

Pros:

• interface is, I think, a little cleaner
• plays nice with subclassing:

class Intermediate extends HTMLElement {
  #internals;
  constructor({ internals = {} }) {
    super({ internals });
    this.#internals = internals;
  }
}

class FinalDerived extends Intermediate {
  #internals;
  constructor() {
    let internals = {}; // note: intentionally not exposing this to any further derived classes. you can instead copy the pattern in Intermediate precisely, if you want to allow further extension.
    super({ internals });
    this.#internals = internals;
  }
}

Cons:

• I don't know if you have the freedom or willingness to add parameters to the HTMLElement constructor
• this sort of C++-style "output parameter" is not familiar to most JS programmers (though the callback style probably would be)
• for any class which allows passing in internals - anything subclassable as in the previous point, or which omits the constructor entirely - any code which constructructs the class can get access to the internal state of the resulting object.

That last point is kind of an interesting one - in the design of this feature, is it acceptable if anyone who constructs a Foo has access to that particular instance of Foo's internal state? (I don't think it's avoidable if you allow further subclassing, because constructing a Foo is not distinguishable to subclassing Foo and constructing the subclass.)

[domenic]

Yeah, that was one of the alternatives we considered: WICG/webcomponents#758 (comment). The general thinking was that it was "too weird", but if TC39 told us it was the right way to do protected, that would certainly change folks' minds.

in the design of this feature, is it acceptable if anyone who constructs a Foo has access to that particular instance of Foo's internal state? (I don't think it's avoidable if you allow further subclassing, because constructing a Foo is not distinguishable to subclassing Foo and constructing the subclass.)

I think the answer is "we would prefer to avoid it if possible". Thus the design where getInternals() can only be called once. That at least allows well-designed HTMLElement subclasses to protect from others messing with their internals, by calling this.getInternals() early in their construction.

I agree it seems to be in conflict with allowing further subclassing. Frustrating.

[littledan]

Maybe we can solve this for custom elements specifically, rather than protected data generally--not proving that the access was actually inside the class, but that the access was set up before customElements.define was called on the class (which, presumably, means that it was collaborating with the class, as custom element classes are unlikely to be sitting around for a while before being registered).

// This would be implemented by the browser, but shown here in JS for readability

// Check that the decorator is only called once, and only before customElements.define
let showProtected = new WeakSet();
function registerProtected(constructor) {
  if (showProtected.has(constructor)) throw new TypeError;
  if (constructor is registered as a custom element) throw new TypeError;
  showProtected.add(constructor);
}

class HTMLElement {
  #protectedState = 0;
  
  // Decorator for a field declaration
  static internals(desc) {
    assert(desc.kind === "field");
    assert(desc.placement === "own");
    assert(desc.initializer === undefined);
    assert(typeof desc.key === "object");  // PrivateName
    let constructor;
    function check(receiver) {
      let definition = receiver's custom element definition;
      if (definition's constructor !== constructor) throw new TypeError;
      if (!showProtected.has(constructor)) throw new TypeError;
    }
    return {
      kind: "method",
      placement: "own",
      key: desc.key,
      descriptor: {
        get() {
          check(this);
          return this.#protectedState;
        }
        set(v) {
          check(this);
          this.#protectedState = v;
        }
      }
      finisher(klass) {
        registerProtected(klass);
        constructor = klass;
      }
    };
  }
}

// This is the subclass
class CustomElement extends HTMLElement { 
  @HTMLElement.internals #internals;

  // Public methods can use this.#internals as appropriate
  increment() {
    this.#internals = this.#internals + 1;
  }
}
customElements.define(CustomElement, "custom-element");

const ce = new CustomElement();
ce.increment(); // will manipulate the private state

Maybe this solution could work for any class which would need to be passed to some sort of function before it's instantiated (e.g., it's required to use some kind of class decorator, c.f. tc39/proposal-decorators#161), but not for arbitrary subclasses.

[saambarati]

Can y'all explain why protected state doesn't work out of the box for this type of thing? It feels like a failure of that proposal that it doesn't.

[ljharb]

@saambarati the proposal is for private and public state; not for "protected" state.

[saambarati]

@ljharb Sorry, I meant the private state proposal. It feels like this is something that should be built into that proposal, which is to have fields only visible to things in the class hierarchy.

[ljharb]

@saambarati See tc39/proposal-private-fields#11; protected has consistently been considered a possible follow-on proposal (that many are skeptical will ever be viable as a first-class mechanism, myself included), and can be handled with decorators in ways that (if i recall) https://github.com/tc39/proposal-private-fields/blob/master/DECORATORS.md#protected-style-state-through-decorators-on-private-state documents.

[saambarati]

@ljharb Thanks for sharing that.

Why are you (and others) skeptical that there can’t be a first class solution here?

I understand the limitations of protected state in a language dynamic in the ways JS is. However, this does really feel like something useful and worth having. It’d be nice if a user could guarantee that once a particular object is created, it has certain fields only accessible from within its class hierarchy.

FWIW, from the alternatives proposed here, C++ style out params feels the least clunky to me even though it’s not a common pattern in JS APIs.

[littledan]

There are several reasons why the class fields proposal doesn't include protected state (apologies for the length of this explanation).

• It's not clear how protected fields could actually prevent their usage from outside the class, given how JavaScript methods can be redirected to be called against an instance of another class. It's not clear to me what sort of semantics we could adopt for protected which would prevent the following "attack":

// Actually built-in to the browser
class HTMLElement {
  protected #internals;
}

// Module defining an encapsulated component
class Component extends HTMLElement { /* ... no use of #internals ... */ }
customElements.define('comp-onent', Component);

// Another module that's not supposed to know about Component's #internals
let el = document.createElement('comp-onent');
class Attacker extends HTMLElement {
  getInternals() { return this.#internals; }
}
let internals = Attacker.getInternals.call(el);  // This just works!

• JavaScript strict mode makes it so that all of the scopes defined within strict mode are nested sort of statically: You can see all of the variable declarations that will affect inner code (unlike sloppy mode, which permits declarations leaking eval as well as with). This property is really helpful for JS front-ends, who need to work out what's happening in scopes to enable the rest of the JIT to do much of anything (though sloppy mode isn't fatal, as you can statically see those bad patterns).
  Private fields are based on a lexically scoped mapping from the name (like #internal) to an internal "private name", which is sort of like the property key used to look up the value in the instance (or, you could think of the private name as the WeakMap that maps instances to their private field value).
  Protected would be sort of like using a with statement, in that this lexically scoped mapping would be extended by whatever private name declarations the superclass had. This would be unfortunate!
• Some TC39 delegates see protected state as generally an anti-pattern, and don't want to encourage its use in JavaScript. (I'd count myself among those who are skeptical that it's the right default, but don't take as strong position as some others.)

Note, in the class fields repository, many people have raised the issue that protected state would need a different sigil other than #. I don't understand this concern, personally; I think we could reuse # for protected, if there were some sort of resolution to the above three issues. However, we need some kind of different syntax at the access site, not just the definition site, as explained in the FAQ.

[waldemarhorwat]

littledan covered the main reasons above. In addition, we'd prefer to have a simple solution, and adding protected would significantly increase complexity.

[getify]

I'm not at all in favor of adding "protected" visibility for properties, but, playing devil's advocate... without it, one of the most important concepts underlying class inheritance -- polymorphism and method overriding -- is rather neutered.

Promises are a perfect example. Subclassing a promise but not being able to access its private slots severely restricts what things you can do in a promise subclass. The internal state of a promise is exactly the kind of state that should actually be "protected" instead of "private", so that a subclass could extend behavior around it.

Unfortunately, protected visibility is sort of antithetical to the concept of a prototypal inheritance -- where everything is either public (and inherited) or private (lexical, etc) and not. It is strange to me that we added extends and super and didn't add the critical connection that makes those two most useful in many class-oriented patterns.

[trusktr]

let internals = Attacker.getInternals.call(el); // This just works!

@littledan Did you mean Attacker.prototype.getInternals?

But no, that shouldn't work. It would be the same as calling el.#internals which shouldn't work, and the spec should prevent that from working.

It's not clear how protected fields could actually prevent their usage from outside the class

In lowclass there's no way the attacker can do that, because the protected and private prototypes are not publicly accessible on the constructor.

Try lowclass (see the fairly extensive tests as documentation) and let me know if you can figure out a way to access the protected members outside the class. I would be very interested to know if there's a way.

Maybe someone can do some very crazy monkey patching of window.WeakMap (good luck to them trying to figure which objects are prototypes or instances of which classes!), but with a native implementation the monkey patching will not be possible.

[trusktr]

adding protected would significantly increase complexity

@waldemarhorwat Are you sure it would be that significant? My whole lowclass implementation (src/index.js, and not including the features in src/native.js which the ES6+ JS engines already supply) is only ~780 lines, and the code there is not very dense and there's lots of comments.

[devsnek]

@trusktr it seems your library actually runs into the same problems...

const T = lowclass(({ Protected }) => class {
  constructor() {
    Protected(this).v = 'T';
  }
});

const t = new T();

function extractTheDeets(V) {
  const S = lowclass(({ Protected }) => class extends V.constructor {
    constructor() {
      super();
      const original = Object.getPrototypeOf(V);
      Object.setPrototypeOf(V, S.prototype);
      const p = Protected(V);
      Object.setPrototypeOf(V, original);
      return p;
    }
  });
  return new S();
}

extractTheDeets(t);

for those interested in how lowclass works, it looks kinda like this:

const weak = new WeakMap();

function hasInstance(O, C) {
  if (typeof O !== 'object' && typeof O !== 'function') {
    return false;
  }
  const P = C.prototype;
  if (typeof P !== 'object' && typeof P !== 'function') {
    throw new TypeError();
  }
  while (true) {
    O = Object.getPrototypeOf(O);
    if (O === null) {
      return false;
    }
    if (P === O) {
      return true;
    }
  }
}

function lowclass(fn) {
  const Class = fn({
    Protected(instance) {
      if (hasInstance(instance, Class)) {
        if (weak.has(instance)) {
          return weak.get(instance);
        }
        const s = {};
        weak.set(instance, s);
        return s;
      }
      throw new TypeError('invalid access');
    },
  });
  
  return Class;
}

[trusktr]

@devsnek Nice, glad to receive some input on it! That trick works with Protected but not with Private.

I could fix it with a monkey patch of Object.get/setPrototypeOf and the __proto__ descriptor and lock the descriptors with low chance of breaking lowclass-consuming applications (which at the moment are only my apps 😄).

I could also delete the .constructor prop, which I usually never need. But sometimes I do need to read static members of an instance, so that's where foo.constructor.staticProp is useful.

Another thing I can do is provide a Static helper for accessing static members without revealing the class constructor. I don't think I've ever needed to access the .constructor prop other than for static member access. I can't remember if I've ever needed it for something else.

The issue would be entirely nonexistent in the native implementation.

[devsnek]

the only alternative would be an internal chain of inheritance, but that seems like a confusing path to take, given that there is already a visible chain of inheritance via prototypes.

[getify]

@devsnek -- there was a TC39 proposal several years back which was basically copy-on-write of "inherited" private slots. So the parent has a private slot with a value, child gets copy of it at instantiation, then every time it's changed in parent, the new value is copied to the child (and vice versa). I was pretty weary of that idea, because it creates an entirely separate path for data to flow from object to object. I guess you could label that copy-on-write as "an internal chain of inheritance". But I'm pretty sure something like that is the only likely approach for private visibility.

[trusktr]

@getify Another way to do it is "access mode" or "reference mode". Under this idea, private and protected fields all simply live on the one prototype chain, are a interoperable with all existing libs (f.e. lodash, underscore, etc).

I'm sure there's other ways! I don't like the current private fields, and I think it is premature for it to be in stage 3. Private fields jumped from stage 1 to stage 3 in 2 months.

[trusktr]

@devsnek Given that JavaScript engines have the advantage that they can track information while keeping it inaccessible from JavaScript, wouldn't you say that my lowclass implementation of protected proves that a JavaScript engine could implement it without the problem you pointed out?

[rdking]

Something just occurred to me, and I feel kind of stupid for not thinking of it sooner. The first issue @littledan listed here, and that @ljharb has had a field day throwing at me every time I mention how much people are going to ask for "protected" support, can be solved just by giving protected members a different reified private name in each subclass.

Isn't that the main reason why the example in the OP isn't immediately disagreeable? If a "protected" declaration translated the field into a field accessor (don't ask me how that would work) that bound itself to the field of the base with the same non-reified name, leaking of the type described would be impossible and the OP would have a simple, non-convoluted means of shared access.

[nicolo-ribaudo]

Do you mean something like this but with specific syntax instead of a decorator?

import { Protected, Inherit } from "./protected";

class Base {
  @Protected #foo = 2;
  
  setFoo(x) { this.#foo = x; }
}

class Child extends Base {
  @Inherit #foo;
  
  getFoo() { return this.#foo }
}

var privateNames = new WeakMap();
/* This weakmap will be like this:
    WeakMap {
      Base => Map { "foo" => Base's #foo }
    }
*/

export function Protected(desc) {
  const { key } = desc;

  desc.extras = [{
    kind: "hook",
    placement: "static",
    finish(Class) {
      if (!privateNames.has(Class)) privateNames.set(Class, new Map);
      privateNames.get(Class).set(key.description(), key);
    }
  }];
}

export function Inherit(desc) {
  const { key, placement, kind } = desc;
  return {
    key, placement, kind,
    get() {
      return getSuperPN(this, key).get(this);
    },
    set(v) {
      getSuperPN(this, key).set(this, v);
    }
  };
}

function getSuperPN(key, instance) {
  return privateNames.get(instance.__proto__.constructor).get(key.description());
}

[rdking]

@nicolo-ribaudo Something like that, yes. Done something like that, each subclass would still never have access to the private names embedded in the parent class, but nobody want's those anyway. The all-important data would simply be accessible via what is essentially a different field name in the subclasses.

[ljharb]

Given that everyone can import that “./protected” module, I’d still consider that fully public, ftr.

[rdking]

@ljharb Did you miss that this example is just a decorators version of an in-engine implementation?

[WebReflection]

Given that everyone can import that “./protected” module, I’d still consider that fully public, ftr.

I think you are ignoring decades of OOP ability to extend classes with protected properties ... protected means by class definition/extend/declaration, so that nobody can access, through the public instance reference, that property ... but I kinda believe you know this already, right?

[domenic]

Since this thread seems to have turned into something else, which I don't have time to keep track of, I'll close it. Thanks TC39 for all the discussion; we'll be going with a design similar to the OP, which you can see in whatwg/html#4324. We're hopeful that if decorators advance we can upgrade to a solution based on them at that time.

[trusktr]

@domenic In the OP, how does a sub class of CustomElement get the internals? Do we have to wait for decorators for this to be convenient?

@devsnek I solved the problem you pointed out in lowclass (that by modifying an object's prototype you were able to trick it to getting a protected member from public scope) by making a simple helper called lockPrototype that makes an object's prototype permanent, but everything else about the object works the same (add/change/configure new and existing properties).

I think lockPrototype is a useful tool, and it'd be nice if it were added to the language as something like Object.lockPrototype (less restrictive than Object.preventExtensions).

An engine would be able to prevent protected access without having to lock a prototype though, though if a prototype is modified, it'd be strange for protected to access members of a prototype which is no longer in the prototype chain, though the same thing is true about super and moving functions around to different prototypes.

And plus @rdking found a way to implement true protected (without modifying prototype configurability) in javascript (hopefully there's no flaws that haven't been found yet).

We know it is possible for the engine though (lexical scope checking is easy), so I simply believe it would be nice to add syntax space for protected, not just for private, even if the protected feature is not added yet.

[ljharb]

@trusktr see https://github.com/tc39/proposal-freeze-prototype

[trusktr]

@ljharb Ah, that's cool. In my case, I needed to make instance prototypes "non-configurable", which I monkey-patched in my previous comment. A native implementation won't need to do this.

@devsnek Can you find another way besides prototype-swapping to leak the protected members?