Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

i18n: introduce script to swap in new locale to LHR #8755

Merged
merged 17 commits into from
Jun 25, 2019
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 20 additions & 4 deletions lighthouse-core/lib/i18n/i18n.js
Original file line number Diff line number Diff line change
Expand Up @@ -168,20 +168,21 @@ const _icuMessageInstanceMap = new Map();
*
* @param {LH.Locale} locale
* @param {string} icuMessageId
* @param {string} icuMessage
* @param {string=} fallbackMessage
* @param {*} [values]
* @return {{formattedString: string, icuMessage: string}}
*/
function _formatIcuMessage(locale, icuMessageId, icuMessage, values) {
function _formatIcuMessage(locale, icuMessageId, fallbackMessage, values) {
const localeMessages = LOCALES[locale];
const localeMessage = localeMessages[icuMessageId] && localeMessages[icuMessageId].message;
// fallback to the original english message if we couldn't find a message in the specified locale
// better to have an english message than no message at all, in some number cases it won't even matter
const messageForMessageFormat = localeMessage || icuMessage;
const messageForMessageFormat = localeMessage || fallbackMessage;
if (messageForMessageFormat === undefined) throw new Error('No ICU message string to format');
// when using accented english, force the use of a different locale for number formatting
const localeForMessageFormat = locale === 'en-XA' ? 'de-DE' : locale;
// pre-process values for the message format like KB and milliseconds
const valuesForMessageFormat = _preprocessMessageValues(icuMessage, values);
const valuesForMessageFormat = _preprocessMessageValues(messageForMessageFormat, values);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@patrickhulce note this change.. it was using the fallback message instead of the one pulled from the locales files. which seemed odd. right?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it was using the fallback message instead of the one pulled from the locales files. which seemed odd. right?

It is weird. As long as there isn't an old mismatched translation for the locale it shouldn't matter, but agreed that checking that the values will actually be able to go into the string we want them to (and preparing them to do so) is the right thing to do.

Mismatched translations could become a problem at some point. If we've updated a string in en-US.json and it has different values than the not-yet-updated strings in all the other locales, I'm pretty sure that will either throw in _preprocessMessageValues or below in the formatter.

Maybe we should have a check in string collection that deletes strings in other locales if the expected values don't match anymore.


const formatter = new MessageFormat(messageForMessageFormat, localeForMessageFormat, formats);
const formattedString = formatter.format(valuesForMessageFormat);
Expand Down Expand Up @@ -277,6 +278,20 @@ function getFormatted(icuMessageIdOrRawString, locale) {
return icuMessageIdOrRawString;
}

/**
* @param {LH.Locale} locale
* @param {string} icuMessageId
* @param {*} [values]
* @return {string}
*/
function formatMessageFromIdWithValues(locale, icuMessageId, values) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about

Suggested change
function formatMessageFromIdWithValues(locale, icuMessageId, values) {
function getFormattedFromIdAndValues(locale, icuMessageId, values) {

to match getFormatted (param order for getFormatted vs _formatIcuMessage is unfortunate but whatever)

const icuMessageIdRegex = /(.* \| .*)$/;
if (!icuMessageIdRegex.test(icuMessageId)) throw new Error('This is not an ICU message ID');

const {formattedString} = _formatIcuMessage(locale, icuMessageId, undefined, values);
return formattedString;
}

/**
* @param {string} icuMessageInstanceId
* @param {LH.Locale} locale
Expand Down Expand Up @@ -349,6 +364,7 @@ module.exports = {
getRendererFormattedStrings,
createMessageInstanceIdFn,
getFormatted,
formatMessageFromIdWithValues,
replaceIcuMessageInstanceIds,
isIcuMessage,
};
84 changes: 84 additions & 0 deletions lighthouse-core/lib/i18n/swap-locale.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
/**
* @license Copyright 2019 Google Inc. All Rights Reserved.
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
*/
'use strict';

/* eslint-disable no-console, max-len */
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably shouldn't leave these on if this is for real now :)

maybe we have a script version that does the console.loging?


const _set = require('lodash.set');
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😍


const i18n = require('./i18n.js');

/**
* @fileoverview Use the lhr.i18n.icuMessagePaths object to change locales
*
* `icuMessagePaths` is an object keyed by `icuMessageId`s. Within each is either
* 1) an array of strings, which are just object paths to where that message is used in the LHR
* 2) an array of `LH.I18NMessageValuesEntry`s which include both a `path` and a `values` object
* which will be used in the replacement within `i18n._formatIcuMessage()`
*
* An example:
* "icuMessagePaths": {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: unindent this 2 spaces so it's easy to tell its nested? :)

"lighthouse-core/audits/metrics/first-contentful-paint.js | title": [
"audits[first-contentful-paint].title"
],
"lighthouse-core/audits/time-to-first-byte.js | displayValue": [
{
"values": {
"timeInMs": 570.5630000000001
},
"path": "audits[time-to-first-byte].displayValue"
}
],
"lighthouse-core/lib/i18n/i18n.js | columnTimeSpent": [
"audits[mainthread-work-breakdown].details.headings[1].text",
"audits[network-rtt].details.headings[1].text",
"audits[network-server-latency].details.headings[1].text"
],
...
*/

/**
* Replaces all strings within an LHR with ones from a different locale
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this might be worth being the place where we explain how the icuMessagePaths property is setup and meant to be used?

* @param {LH.Result} lhr
* @param {LH.Locale} requestedLocale
* @return {LH.Result}
*/
function swapLocale(lhr, requestedLocale) {
// copy LHR to avoid mutating provided LHR
lhr = JSON.parse(JSON.stringify(lhr));

const locale = i18n.lookupLocale(requestedLocale);
const {icuMessagePaths} = lhr.i18n;

Object.entries(icuMessagePaths).forEach(([icuMessageId, messageInstancesInLHR]) => {
for (const instance of messageInstancesInLHR) {
// The path that _formatPathAsString() generated
let path;

This comment was marked as resolved.

let values;
if (typeof instance === 'string') {
path = instance;
} else {
path = /** @type {LH.I18NMessageValuesEntry} */ (instance).path;
// `values` are the string template values to be used. eg. `values: {wastedBytes: 9028}`
values = /** @type {LH.I18NMessageValuesEntry} */ (instance).values;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't need these casts anymore

}
// If we couldn't find the new replacement message, keep things as is.
try {
// Get new formatted strings in revised locale
const formattedStr = i18n.formatMessageFromIdWithValues(locale, icuMessageId, values);
// Write string back into the LHR
_set(lhr, path, formattedStr);
} catch (e) {}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unconditional catch here seems excessive since all problems would be silenced. Some wouldn't be interesting (e.g. string from removed audit isn't available in new locale), but some could be a real problem. Maybe at least log the issue?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe at least log the issue?

or log the ones that aren't 'No ICU message string to format' if that gets excessive

}
});

lhr.i18n.rendererFormattedStrings = i18n.getRendererFormattedStrings(locale);
// Tweak the config locale
lhr.configSettings.locale = locale;
return lhr;
}

module.exports = swapLocale;
33 changes: 33 additions & 0 deletions lighthouse-core/test/lib/i18n/swap-locale-test.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
/**
* @license Copyright 2019 Google Inc. All Rights Reserved.
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
*/
'use strict';

const swapLocale = require('../../../lib/i18n/swap-locale.js');

const lhr = require('../../results/sample_v2.json');

/* eslint-env jest */

describe('swap-locale', () => {

This comment was marked as resolved.

it('can change golden LHR english strings into spanish', () => {
const lhrEn = /** @type {LH.Result} */ (JSON.parse(JSON.stringify(lhr)));

const lhrEs = swapLocale(lhrEn, 'es');

// Basic replacement
expect(lhrEn.audits.plugins.title).toEqual('Document avoids plugins');
expect(lhrEs.audits.plugins.title).toEqual('El documento no usa complementos');

// With ICU string argument values
expect(lhrEn.audits['dom-size'].displayValue).toEqual('31 elements');
expect(lhrEs.audits['dom-size'].displayValue).toEqual('31 elementos');

/* eslint-disable max-len */
// Renderer formatted strings
expect(lhrEn.i18n.rendererFormattedStrings.notApplicableAuditsGroupTitle).toEqual('Not applicable');
expect(lhrEs.i18n.rendererFormattedStrings.notApplicableAuditsGroupTitle).toEqual('No aplicable');
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't even notice this was translated 😆 is there a better example?

});

This comment was marked as outdated.

});
2 changes: 2 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@
"@types/jest": "^24.0.9",
"@types/jpeg-js": "^0.3.0",
"@types/lodash.isequal": "^4.5.2",
"@types/lodash.set": "^4.3.6",
"@types/make-dir": "^1.0.3",
"@types/mkdirp": "^0.5.2",
"@types/node": "*",
Expand Down Expand Up @@ -119,6 +120,7 @@
"isomorphic-fetch": "^2.2.1",
"jest": "^24.3.0",
"jsdom": "^12.2.0",
"lodash.set": "^4.3.2",
"make-dir": "^1.3.0",
"npm-run-posix-or-windows": "^2.0.2",
"nyc": "^13.3.0",
Expand Down
3 changes: 2 additions & 1 deletion types/lhr.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ import LHError = require('../lighthouse-core/lib/lh-error.js');

declare global {
module LH {
export type I18NMessageEntry = string | {path: string, values: any};
export type I18NMessageValuesEntry = {path: string, values: any};
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we should type values as Record<string, string | number>?

export type I18NMessageEntry = string | I18NMessageValuesEntry;

export interface I18NMessages {
[icuMessageId: string]: I18NMessageEntry[];
Expand Down
12 changes: 12 additions & 0 deletions yarn.lock
Original file line number Diff line number Diff line change
Expand Up @@ -528,6 +528,13 @@
dependencies:
"@types/lodash" "*"

"@types/lodash.set@^4.3.6":
version "4.3.6"
resolved "https://registry.yarnpkg.com/@types/lodash.set/-/lodash.set-4.3.6.tgz#33e635c2323f855359225df6a5c8c6f1f1908264"
integrity sha512-ZeGDDlnRYTvS31Laij0RsSaguIUSBTYIlJFKL3vm3T2OAZAQj2YpSvVWJc0WiG4jqg9fGX6PAPGvDqBcHfSgFg==
dependencies:
"@types/lodash" "*"

"@types/lodash@*":
version "4.14.106"
resolved "https://registry.yarnpkg.com/@types/lodash/-/lodash-4.14.106.tgz#6093e9a02aa567ddecfe9afadca89e53e5dce4dd"
Expand Down Expand Up @@ -5584,6 +5591,11 @@ lodash.memoize@~3.0.3:
resolved "https://registry.yarnpkg.com/lodash.memoize/-/lodash.memoize-3.0.4.tgz#2dcbd2c287cbc0a55cc42328bd0c736150d53e3f"
integrity sha1-LcvSwofLwKVcxCMovQxzYVDVPj8=

lodash.set@^4.3.2:
version "4.3.2"
resolved "https://registry.yarnpkg.com/lodash.set/-/lodash.set-4.3.2.tgz#d8757b1da807dde24816b0d6a84bea1a76230b23"
integrity sha1-2HV7HagH3eJIFrDWqEvqGnYjCyM=

lodash.sortby@^4.7.0:
version "4.7.0"
resolved "https://registry.yarnpkg.com/lodash.sortby/-/lodash.sortby-4.7.0.tgz#edd14c824e2cc9c1e0b0a1b42bb5210516a42438"
Expand Down