feat(core): ldml processor touch support 🙀

[srl295]

@mcdurdin this is somewhat of an umbrella ticket. Trying to capture the discussion in issues for planning.

Support Touch Layouts in LDML.

• Waiting on Joshua's 📜 merges

• WASM working well, but need more WASM API endpoints into Core

• main blocker: Q: how to implement core into KMFW?

• Divergence in other platforms

• May be able to do some test scaffolding for touch (prior to KMFW intergration)

• Q: how to link to processEvent in a lightweight way but without breaking predictive text?

• Note: Core in KMFW is not in v17.

• 🙀 spec(core): LDML KeyboardProcessor 🙀  #5015

[jahorton]

• Q: how to link to processEvent in a lightweight way but without breaking predictive text?

Sounds like it's time for a whirlwind tour of common/web/keyboard-processor.

OutputTarget

common/web/keyboard-processor/src/text/outputTarget.ts

This is the common base class for all forms of context supported by the Keyman Engine for Web. It includes deadkey / marker tracking:

keyman/common/web/keyboard-processor/src/text/outputTarget.ts

Lines 83 to 85 in b4df4ab

	deadkeys(): DeadkeyTracker {
	return this._dks;
	}

Even predictive-text context extends this base class: the Mock class from the same file supports fully synthetic contexts and context 'cloning'.

The per-deadkey struct:

keyman/common/web/keyboard-processor/src/text/deadkeys.ts

Lines 2 to 6 in b4df4ab

	export class Deadkey {
	p: number; // Position of deadkey
	d: number; // Numerical id of the deadkey
	o: number; // Ordinal value of the deadkey (resolves same-place conflicts)
	matched: number;

ordinal: if two deadkeys are in the same "virtual" position, this field resolves which came first.
matched: used during keyboard-rule processing.

Note that we do NOT actually insert the deadkeys into the raw text-context! Web's pattern here is different. More on that later in this comment; we do have a deadkey-inlining method that may make a useful reference.

KeyEvent

common/web/keyboard-processor/src/text/keyEvent.ts

Any and all forms of keystroke events are preprocessed into instances of that class; these instances are then passed into the Web engine's version of the KeyboardProcessor, likely similar to what would exist for core.

Note that for fat-finger correction, we pass key events that weren't pressed into the rule processing engine as well, backed by a Mock-cloned context copy.

isSynthetic

isSynthetic - true if generated through interaction with an on-screen-keyboard or for fat-finger simulation. Only false if sending the basic key-event data through to the destination, without rule processing, leads to default handling picking up the slack.

For applications of false, consider a hardware K_ENTER keystroke - this could be interpreted as a "submit form" / "search" command depending on the context.

When true, we simulate the results of "standard" default key-event handling because nothing else can handle it. At present, we don't attempt to force / synthesize an event that could trigger the "true" default.

KeyboardProcessor

keyman/common/web/keyboard-processor/src/text/keyboardProcessor.ts

Line 217 in b4df4ab

processKeystroke(keyEvent: KeyEvent, outputTarget: OutputTarget): RuleBehavior {

The method linked above is the primary entrypoint for rule processing of individual, preprocessed KeyEvents. I doubt you'll need to inspect its internals much. It's named after Core's equivalent class, as it pretty much fills the same role. I'm almost certain this is the top-level class that an LDML engine would need to replace.

Noting the specified return type...

RuleBehavior

common/web/keyboard-processor/src/text/ruleBehavior.ts

This fellow provides a metadata perspective on what the effects of keyboard-rule processing for the KeyEvent were. As Keyman keyboards do include a few aspects of state - "variable stores", in particular - we have to keep those from taking effect for predictive-text keystrokes. They're "committed" by the next layer up from that "primary entrypoint" function, which knows which key events are real and which ones are synthesized fat-finger simulation events.

The predictionPromise field is set by that "next layer up" as well, not by keyboard rule processing.

triggerKeyDefault relates to how we handle things like word-final sigma in Greek.

I believe that it also relates to things like simple K_ENTER presses from hardware keyboards, since K_ENTER can often serve as a command-key in certain contexts. (We don't yet handle changing the intent of the OSK K_ENTER.) When embedded in the mobile apps, the mobile apps know the true context, and not the Web engine, so we rely on the host-app to make the final call on how to interpret such a keystroke. History on this aspect can be found at #5553 and its links should it prove relevant. (If it does, this'd probably be the trickiest aspect to deal with for integration with the Web engine.)

Predictive-text integration

The next layer up from KeyboardProcessor is currently named InputProcessor.

common/web/input-processor/src/text/inputProcessor.ts

This layer is the primary predictive-text driver, responsible for fat-finger emulation and forwarding of new post-keystroke context to the predictive-text engine.

This is the method that most directly receives all KeyEvent instances before forwarding them to KeyboardProcessor:

keyman/common/web/input-processor/src/text/inputProcessor.ts

Line 118 in b4df4ab

private _processKeyEvent(keyEvent: KeyEvent, outputTarget: OutputTarget): RuleBehavior {

It calls this one in order to do fat-finger simulation:

keyman/common/web/input-processor/src/text/inputProcessor.ts

Line 246 in b4df4ab

private buildAlternates(ruleBehavior: RuleBehavior, keyEvent: KeyEvent, preInputMock: Mock): Alternate[] {

Both directly include calls to the primary KeyboardProcessor instance.

On inlining deadkeys

You may find this method to be of use in regard to deadkeys:

keyman/common/web/keyboard-processor/src/text/kbdInterface.ts

Lines 354 to 362 in b4df4ab

	/**
	* Builds the *cached or uncached* keyboard context for a specified range, relative to caret
	*
	* @param {number} n Number of characters to move back from caret
	* @param {number} ln Number of characters to return
	* @param {Object} Pelem Element to work with (must be currently focused element)
	* @return {Array} Context array (of strings and numbers)
	*/
	private _BuildExtendedContext(n: number, ln: number, outputTarget: OutputTarget): CachedExEntry {

We use this to create the 'inlined' equivalent for keyboard rules that need it, broken down into array form. For a quick example used in unit testing...

keyman/common/web/keyboard-processor/tests/node/engine/context.js

Lines 141 to 147 in b4df4ab

	// Match condition for rule
	rule: ['a', {t:'d', d: 0}, {t:'d', d: 1}, 'b'],
	// Start of context relative to cursor
	n: 5,
	ln: 4,
	// Resulting context map
	contextCache: ['a', 0, 1, 'b'],

{t: 'd', ...} says "match a deadkey" here. contextCache is what the _BuildExtendedContext function will produce for a context matching the loose rule definition.

• Visible string: ab
• The two deadkeys (id 0, id 1) exist between a and b in that specific order.

Other notes

One aspect of predictive-text that might be relevant, but probably isn't: application of a predictive-text suggestion or reversion is capable of rewinding the context to restore the "correct" context that it matches. The Transcription class (also in common/web/keyboard-processor/src/text/outputTarget.ts) is at the core of this mechanism.

[mcdurdin]

#8093 (comment) could be added to web internal docs

[ermshiperete]

@mcdurdin Since I won't get to this in the current sprint I'll move it to the end of the scheduled web-core work.

[mcdurdin]

This is a key part of web-core work; moving to S19 to review after Christmas