move to new SES #1201
Merged
move to new SES #1201
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
warner
force-pushed
the
477-new-ses-8
branch
8 times, most recently
from
eca4b45 to
1f26a03
Compare
|
Ok, at long last, all the tests are passing, and this is open for review: There is pretty much one patch per package, and this touches most of them (some in small ways, others quite severe). I'd like to assign the following components for review, please check the box when you're done:
(@erights @dtribble @dckc of course feel free to chime in on anything that catches your attention) I've added a branch to https://github.com/Agoric/dapp-encouragement named Some negative consequences of this PR:
Some good things about this PR:
|
warner
requested review from
michaelfig,
Chris-Hibbert,
katelynsills,
FUDCo and
kriskowal
warner
added
SwingSet
Some ERTP unit tests now use `@agoric/install-ses` to create a SES environment, rather than `@agoric/evaluate` (which is being removed).
The 'agoric deploy' command runs ther dapp's deploy scripts, giving them captp-style access (i.e. `E(zoe).install()`) to objects on the chain and/or the local ag-solo. Previously these were loaded with '@agoric/evaluate', mostly so we could apply the tildot transform to them. Rather than switch to using `import-bundle` and defining a suitable confined environment for the deploy scripts, we now import them unconfined, with a plain `require()` call (which can be replaced by a dynamic import expression when we switch fully to ESM). This allows them to use `fs` and other Start-Compartment authorities (including unrestricted Date.now and Math.random, if it really wanted them). We can tell devs that these are normal JS programs (with captp/E added), rather than needing to explain the ways in which they are not. As a result, the `agoric` tool does *not* need to run under SES. We do not provide the tildot transform. Deploy scripts must use `E()` rather than tildot.
The bundle-source unit tests now create a SES environment with `@agoric/install-ses`, rather than using `@agoric/evaluate`. This should not affect downstream uses.
Programs which need both metered globals and SES should do `import '@agoric/install-metering-and-ses'` before any other imports.
This used to provide a three-argument evaluator function that worked (safely) under SES and worked (unsafely) outside of SES. Our new approach is to require SES all the time, and use either `import-bundle` to evaluate entire module graphs, or the `Compartment` API directly for evaluation of simple strings.
Now that `@agoric/evaluate` is gone, bundle-source no longer needs to know that imports of it are treated specially.
ses-adapter used to provide a best-effort `harden` in non-SES environments. It aspired to provide `Compartment` as well, but that proved too difficult. Our new approach is to require a SES environment, and get `harden` and `Compartment` from the global.
This removes `@agoric/transform-eventual-send/src/rewriter`, which was used to build a tildot transformation function that fit into the old-SES Realm `transforms` option. Now that we're using new SES, it is no longer needed. The unit test which exercised it was removed too, in favor of one that simply tests the core string-to-string transform function. This also removes the `yarn build` step, which used `rollup` to create a single-file distribution bundle. We're removing these bundle steps across the codebase.
Some packages try to modify properties that SES freezes, like `Error.prepareStackTrace`. These packages cannot be imported in a SES environment. Work is underway to tolerate these (endojs/endo#251). `depd` is one such package. As a temporary workaround, this uses the `patch-package` tool to modify the installed `depd` code during the `yarn install` process, patching out its attempt to mutate frozen primordials. When that bug is fixed, we should remove this change.
tame-metering no longer supports old SES, so `SES1ReplaceGlobalMeter` and `SES1TameMeteringShim` have been removed. A new `isTamed` export was added to query whether the global-metering shims have been executed. The taming must be performed before a new-SES `lockdown()`, so application code that runs later cannot install the taming itself. This export helps that application code know whether to bother with additional metering features: if the globals are not tamed, there's not much point in applying source-code-injection taming.
The tests were changed to use `install-ses` to create a SES environment. One test was partially disabled until I figure out how to test the new-SES environment properly (it exercises evaluation of the transformed cde, and was dependent upon getting an old-SES tamed global environment). My plan for this test is to exercise just the source transformation, and have a separate test which exercises evaluation. This should not affect downstream usage.
This commit performs an extensive rewrite of SwingSet's handling of SES, vats, transforms, and metering. SwingSet now requires the host application to provide a SES environment (generally by importing `@agoric/install-ses` or `@agoric/install-metering-and-ses` as the first line of the program). It no longer creates an (old-)SES Realm internally. This increases safety drastically, as we no longer have to carefully prevent primal-realm objects from slipping across the boundary to kernel space. Now there is only one `Object` object in the entire runtime. Each vat gets a separate Compartment via `import-bundle`. The `buildRootObject` function of each vat, which receives arguments like `E` and `D` (to perform eventual-sends and invoke device nodes), also receives a `vatPowers` object. This includes three new properties: `makeGetMeter` (to create a Meter object that the kernel will refill between cranks), `transformMetering` (which can be given to a Compartment constructor to inject metering code), and `transformTildot` (which supports the use of concise eventual-send syntax in evaluated code, e.g. a REPL). Vats are not metered yet, but enough support code and tests have been added to make the process easier in the future. If the kernel finds "metered globals" support (which happens in applications that import `@agoric/install-metering-and-ses` when they begin), the kernel will use it to improve metering protection for vat code that uses it. The "bundle kernel source" step has been removed. `buildVatController` uses `bundle-source` (which uses `rollup`) at runtime to turn the kernel source graph into something that can be evaluated in a new Compartment, and then `import-bundle` to do the evaluation.
The swingset-runner application now installs SES at startup, to support the new SwingSet library that requires it.
The sharing-service tests which use SwingSet now require a SES environment, which it gets by using `@agoric/install-ses`. The library code itself did not change, so this should not affect downstream usage.
Zoe's contract evaluator was rewritten to use `import-bundle` and the new metering system. This requires a SES environment, so any downstream code which uses zoe must be run in an application that does `import '@agoric/install-ses'` (or `install-metering-and-ses`) at startup. The API has changed slightly: the `install()` method now takes a single argument named `bundle`, whereas previously it took a tuple of `(source, moduleFormat)`. The `bundle` argument should be exactly the output of `await bundleSource(path)`. The internal contents of the bundle are private to `bundle-source` and `import-bundle`: future changes will be made in both simultaneously. However `install()` retains backwards compatibility for the same of external dapps, until we get them all rewritten. The `getInstallation()` method now returns the entire bundle, not just the source code component. When comparing bundles (to make sure the installation points to the code you're expecting), you must do a deep comparison of the bundle objects, rather than a shallow comparison of source code strings. The `makeZoe()` constructor no longer needs to be given `require`, but old dapps (in other repos) might still provide it. We arrange our endowments to make sure a leftover `require` argument won't interfere with the one we create locally, so `makeZoe` is backwards compatible with those dapps. However, if metering is desired, `makeZoe` requires a second argument named `vatPowers` which is normally obtained from the vat's `buildRootObject()` arguments. Zoe will pull `transformMetering` and `makeGetMeter` functions from `vatPowers` to implement metering. If `vatPowers` is not passed through the vat `setup` function and delivered to `makeZoe()`, metering will be disabled.
The 'spawner' is still evolving. At present it is a subset of Zoe: it can evaluate bundles of code in a metered/confined environment, it has "install" (class-ish) and "spawn" (instance-ish), but it lacks Zoe's notion of offer safety and terms and other economic nouns. It includes the backwards-compatibility layer to allow the deploy scripts in external dapps that still use a two-argument `install()` to continue to work, until we rewrite them to use a single `bundle` argument. This updates spawner to use new SES, so programs which include it must provide a SES environment (generally by doing `import '@agoric/install-metering-and-ses'` as their first line). It also uses the new metering approach.
cosmic-swingset contains our two primary applications: `ag-cosmos-chain` and `ag-solo`. These have both been updated to use `install-metering-and-ses` to provide a globally-metered SES environment for SwingSet. The REPL has been updated to use the Compartment API, as well as the new transformTildot function. This adds node-lmdb to package.json, to support the LMDB-vs-SES workaround that requires us to import LMDB before lockdown. When that gets fixed and we remove the import, we should remove the package.json dependency too
This updates yarn.lock with all the changes in this PR. This commit lives at the end to make rebasing easier.
The agoric-cli/integration-test was failing: while dapp-encouragement's contract was being installed into the on-chain Zoe, the zoe.install() crank spends a lot of time in babel applying the metering transform to the contract bundle. Most of this happens during import, after the initial nestedEvaluate wrapper executes, which means the global meter is active. All that babel code calls primordial functions like Object.keys a lot, and all those calls count against the meter. The dapp-encouragement contract required about 12M total units, and the default meter budget only allowed 10M. After raising the meter value to let the test pass, @michaelfig helped me find a way to turn off the global meter while inside the transform functions, so we don't "charge" contracts for their usage of Babel. This reduces the dapp-encouragement evaluation's metering usage by at least a factor of 10, making the budget increase mostly unnecessary. I'm going to leave the framework for it in place for now, however. We might lower the budget back to the original values later, if meter-exhausting tests take too long, but so far they seem to "finish" infinite loops quickly enough.
LMDB is somewhat incompatible with SES: it uses an `Error`-modifying trick to locate the shared library that it needs to dynamically load into the process at startup, and of course SES freezes `Error`. Until SES finds a way to tolerate this, the workaround is to perform a dummy `import 'node-lmdb'` before importing `@agoric/install-ses`. The test now exercises this, but programs that use swingset (and thus swing-store-lmdb) must do it too. This does not affect the API, so it should not be visible to downstream code, however that code *will* see the change in the SwingSet semver number and must update to match (by installing SES at startup).
This was referenced Jun 27, 2020
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I'm not quite ready for review, but it's getting close. This branch passes all existing tests, and only disables a few (transform-metering,Ready for review!test-zzz-eval). I expect to reenable+rewrite that test before I ask for review.#dapp-encouragement-branch: new-ses
refs #477
This PR is broken up into basically one commit per package. It touches most of them.
Starting SES:
import '@agoric/install-ses'In the new world, programs which use SES must begin with
import '@agoric/install-ses'. By "program", I mean anything you run from a shell, or withnode -r esm PROGRAM: the entry point that is not itself imported by anything else. A program "uses SES" if any library transitively included by that program uses SES features likehardenorCompartment. The author of some code might expect (or rely upon) their code to run under SES, but it is the author of the final top-level program who makes that decision.@agoric/install-sesimports and executeslockdown()from the SES-shim; doing this as an import side-effect ensures that all subsequent imports are performed against tamed primordials in the "SES Start Compartment". It does not make sense to calllockdown()twice, which is why the entry point should do it, not every library (by the time the library is loaded, it's too late to calllockdown).We have three such main programs:
ag-solo(a swingset running in a local singleton process),ag-chain-cosmos(a swingset running in a replicated chain node), andswingset-runner(utility launcher for vat execution, still evolving). These all now start withimport '@agoric/install-ses'(or equivalent).We also have a whole bunch of test programs, scattered throughout the monorepo, making up our unit test suite (about 93 of them). Each
test-whatever.jsprogram can be run standalone (node -r esm test/test-whatever.js), and this is a handy way to run a subset of the tests. They can also be invoked by a test runner, such as theyarn testscripts that runtaportapein some form. Roughly half (53) of these now import something that expects SES, and now start with an import of@agoric/install-ses. Becauselockdown()affects the entire containing process, these tests want to be run in separate processes. Therefore the packages which contain SES-using tests have been changed to make theiryarn testscript usetap(which runs each test program in a separate process, sometimes in parallel) rather thantape(which imports all the programs it finds into a single shared process).Any "vetted shims" must be applied before
lockdown()is called. We have one such shim:tame-metering, which modifies the global objects to check (and decrement) a meter before any operation that might lead to memory exhaustion or an infinite loop. This is applied by import side-effects too. Programs which need to provide a metered SES environment must doimport '@agoric/install-metering-and-ses'instead ofinstall-ses. We have 3 unit tests which need this, plus the two defensive swingset hosts (ag-soloandag-chain-cosmos).All SES All the Time
Previously, we supported both SES and non-SES modes: SwingSet created the old SES Realm internally, if asked nicely. We had libraries like
ses-adapterwhich provided semi-functionalhardenandCompartmentobjects when used outside of SES, and@agoric/evaluateto provide evaluation functions with same API whether inside or outside SES.This PR removes
@agoric/evaluate,@agoric/ses-adapter, (and incidentally@agoric/default-evaluate-options). All production programs willinstall-sesat their start and never look back. Test programs which do not strictly need SES can be run without it, and we should be conscious about using SES features in libraries that may be used in other environments (i.e. do we limit their audience by imposing a SES requirement upon the program that finally includes them?), but our new default is SES, all-in.bundle-sourceandimport-bundleWe've been using
@agoric/bundle-sourceto encapsulaterollup, to turn source files on disk into a single string (or at least into a single JSON-serializable aggregate source artifact). That continues. All source that will be loaded into SES Compartments (specifically non-start Compartments) will be processed throughbundleSource. Note thatbundleSourcecan run under SES, but only in the start compartment, as it needs filesystem access, and thebabellibrary it uses does not know how to work without it.The receiving side of these source bundles used to be a pattern of
@agoric/evaluateinvocations which involved an endowment namednestedEvaluate, and a number of switches on the source bundle's "moduleFormat" value. This has been replaced by a package named@agoric/import-bundle. The single-object bundle that is emitted bybundle = await bundleSource(startFilename)is destined to be consumed by a call tonamespace = await importBundle(bundle). TheimportBundlecall has options for setting transforms and endowments, and the more interesting invocations (especially those involving metering) contain some lingering intricacies that will be refactored away in due time.For evaluation of single expressions, rather than bundled module trees, code should create a new
Compartment(which is available as a global in all SES-environment code) and invoke thec.evaluate()method. The REPL does this.bundleSourceapplies the "tildot transform" (convertingx~.foo(arg)intoHandledPromise.applyMethod(x, "foo", [arg])). So the input tobundleSourceis allowed to use the concise eventual-send aka "wavy dot" syntax. For code that needs to accept tildot in dynamically-provided expression strings (i.e. the REPL), a transform function must be provided to theCompartmentconstructor.Any APIs which previously worked with
source, moduleFormattuples (e.g. contractHost) should instead use a singlebundleobject. Thespawner.install()method was made backwards compatible, because we have dapps outside the monorepo which will take a while to get updated.importBundlecurrently usesc.evaluate, but a future version will probably use the upcoming SESc.import(), along with the archive format being developed as part of theendotool.Transforms
We have two transform functions: one for tildot, the other for metering. Both depend upon babel, which is not happy running outside the Start Compartment. SwingSet provides these two functions in a new
vatPowersargument, which is given to each vat'sbuildRootObjectfunction, which can share it to other objects within the vat as it sees fit. These functions are nominally pure, and could be provided as endowments or as special module imports, if we wanted. The best delivery pathway is still up for discussion.To construct the transform functions, we must currently import two items from babel separately, and pass them into the transform constructor. This will be simplified eventually, and the transform packages will export a simple pre-built function.
Metering
We have three places that need to prepare a metered evaluation environment:
zoe/src/evalContractCode.js,spawner/src/contractHost.js, and the kernel's own dynamic vat launcher (which has not yet been updated). The previous approach gave vats access to functions namedsetGlobalMeter(which tells the instrumented primordials to meter their usage) andregisterEndOfCrank(which they used to reset their meters after the crank finished and control was transferred back into the kernel). The new approach leaves the kernel in control of meter resetting, and instead gives vats amakeGetMeter()function: they use this to create thegetMeterendowment for a metered Compartment.I'll write up more details on the new metering approach elsewhere. The short description is that the root object of each vat, via
makeGetMeter(), can ask for a new meter that is either refilled (by the kernel) between each crank, or is never refilled. By applying this to a new Compartment, all code loaded by that compartment will be billed against that meter, and if/when the meter expires, all such code will throw an exception that cannot be caught by that same code.This is enough to cover what we're doing right now (protecting against runaways in contract code), but it is not yet sound (there are ways to escape the meter), nor is it particularly safe (the goal is "death before confusion", but that will have to happen one vat at a time, so we need to put contract instances in their own vat first). The metering system will become much more sophisticated before we're done with it.
Build Steps Removed
All
yarn buildsteps which producedrollup-based files have been removed. With the exception of the React app inwallet-frontendand the Go code incosmic-swingset, it should no longer be necessary to runyarn buildafter making code changes in some upstream package.SES Incompatibility Workarounds
The SES shim (and specification) is still evolving to manage compatibility with popular libraries. This PR introduces several workarounds for compatibility problems:
@agoric/install-sesusesmathTaming: 'unsafe'to tolerateMath.randombeing used by dependencies of thetempmodule incosmic-swingsetdateTaming: 'unsafe'allow swingset hosts to build a timer device with wallclock timeError.prepareStackTraceto locate the binarylmdb.soshared-library file it needs to loadpatch-packageis used to modify thedepdpackage atyarn installtime, to work around a similarError.prepareStackTraceproblemWe expect these to all go away as SES-shim grows to accomodate more libraries. In particular, the Start Compartment will probably get a full-power
DateandMathobject, with the troublesome (non-deterministic) methods merely being removed from all non-Start compartments.SwingSet Compartments
SwingSet now uses
importBundleextensively. The kernel is loaded into its own Compartment (no tildot transform, no metering). Each static vat is put into its own Compartment as well. Dynamic vats are the same, although we expect to impose the metering transform on dynamic vats (but maybe not static vats, not sure yet).I've started a document in
packages/SwingSet/docs/vat-environment.mdto describe the JavaScript environment available to vat code. We're still working out what objects should be made available as globals, special imports, and/or as arguments to thebuildRootObjectfunction. For now, the only import that is special is@agoric/harden: thebundle-sourcepackage treats it as an "external", so the bundle itself will still containrequire('@agoric/harden')invocations (and all other imports will be interpolated into the bundle). The vat environment provides arequireendowment that delivers the globally-availablehardenobject when asked (and rejects all other names).hardenis available globally too, but I decided to refrain from removing the 270 occurrences ofimport harden from '@agoric/harden'for now.Vat
setup/buildRootObjectDefinition FilesWe have an uncomfortable amount of boilerplate in the way we define vats, and this PR adds slightly to that burden when the vat in question needs the ability to meter code. A new
vatPowersobject is involved, and it must be passed fromsetup()'s arguments intomakeLiveSlots, and then it must also be passed through to the actualbuildorbuildRootObjectfunction. This boilerplate is scheduled for refactoring, so trust that this will become simpler soon. For now, if you need to create a vat that will meter code, look atpackages/SwingSet/test/metering/vat-within.jsfor the updates you'll need.Zoe/spawner Changes
As described above, APIs which previously worked with two arguments (
sourceandmoduleFormat) should now only use one (bundle). Zoe has a feature whereby you can install a bundle and get back an "installation handle", and then you can take an alleged handle and get back the source code that was used duringinstall. Previously this returned thesourcestring (omitting themoduleFormatvalue). Now it returns the samebundlethat was passed toinstall. Thebundleis defined to be a JSON-serializable object, but the contents are opaque (the details are a private API betweenbundle-sourceandimport-bundle). As a result, clients who wish to compare the installation details must be prepared to do a deep-comparison operation on the bundle objects, rather than a shallow comparison on source-code strings.The
makeZoe()call no longer requiresevaluateas an endowment, however it does require a second argument, which contains thevatPowersobject passed intobuildRootObject.