Merge openwpm-webext-instrumentation repo #328
Conversation
Created with bitjson/typescript-starter@fd420b8 Disable tslint-immutable, Include VS Code debugging config, Include CircleCI config, Include Travis CI config
(at the time of the merge of #221)
…error reporting for things happening in the background event listener
…ods previously found in loggingdb.js) and adhere to no-this-assignment rule once again
Return the response body as Uint8Array instead of string
A bit more customizable JS instrumentation
…essages before the data receiver has been configured + exposing handleJsInstrumentationMessage
Js instrument listen early
In case that the HTTP POST data cannot be parsed as formData, it is also possible to access it as raw binary data. This was previously not supported by the extension and no HTTP POST data was logged for HTTP POST requests that failed to parse as formData by Firefox. This commit handles those cases as well. The data is made available by Firefox as a list of 2-touples of filename - content pairs. The content is presented in a binary form. Since that content might not represent a valid utf8 string, a base64 encoding is applied so that the content is finally exported as a JSON encoded string. This requires that the receiving code expects a new field named *post_body_raw*.
Handle raw HTTP POST data as well
…into merge-openwpm-webext-instrumentation-repo
This reverts commit a408495.
…ad of branded firefox (which doesn't run web extension experiments)
…code at 2018-11-27 (the version was retroactively tagged and thus the code still contained the old version number)
|
Hi, about the Dockerfile: Could you move both extensions in a top level folder? I'm currently working on an improved Dockerfile and having the extension code in a dedicated top level folder would make things much more beautiful and also easier. My current code is in: https://github.com/eriktews/OpenWPM/blob/dockerfile-dev/Dockerfile#L56 Since Docker doesn't have a "COPY --exclude" feature, having the extensions in a dedicated top level folder would make this part of the Dockerfile much shorter. |
|
@eriktews There is only one extension, but if I understand correctly you' suggest that we move |
I created a PR for that: #332 So right now, I would just suggest to merge your PR as it is since it works with #332 too. |
There was a problem hiding this comment.
We should remove/merge/update these files:
- automation/Extension/webext-instrumentation/.gitignore
- automation/Extension/webext-instrumentation/.travis.yml
- automation/Extension/webext-instrumentation/README.md
It may also make sense to move the editor config and npm ignore files as well (at least up one directory so they apply to both the module and the extension).
| @@ -10,7 +10,7 @@ const defaultConfig = { | |||
| overwriteDest: true, | |||
| }, | |||
| run: { | |||
| firefox: process.env.FIREFOX_BINARY || "firefox", | |||
| firefox: process.env.FIREFOX_BINARY || "firefoxdeveloperedition", | |||
There was a problem hiding this comment.
Why this change? From what I can tell the unbranded build is registered as "Nightly". It seems like we shouldn't need any fallback though, as we really only want to test/crawl with the firefox binary grabbed from our install script and throwing an error seems preferable to silently falling back to the system's firefox or dev edition binary. (If I'm misunderstanding how this is used, lmk).
There was a problem hiding this comment.
I changed this so that one can run npm start and get a working extension up and running in Firefox, rather than a broken experience. It'd be better if this can run the bundled Fx installation, but not sure how to set that up in a cross platform way. Using firefox dev edition to start the extension is at least better than the branded firefox (which doesn't run web extension experiments), but we can default to "nightly" too, albeit it may be less likely that extension developers have Nightly installed.
There was a problem hiding this comment.
(If I'm misunderstanding how this is used, lmk).
process.env.FIREFOX_BINARY allows developers to supply their preference when running npm commands, eg, FIREFOX_BINARY=nightly npm start or FIREFOX_BINARY=/Applications/Firefox.app/Contents/MacOS/firefox-bin npm start.
With the idea of later splitting back this code into it's own repository, I suggest we keep some of these files:
I'd like to keep this one, since it is specific for the webext-instrumentation folder and should not be coupled with the framework gitignore rules.
This one can be removed.
I'd like to keep a README / set of docs specific to this folder rather than merging with the framework docs. Again, it makes it easier to split later.
Npm ignore should be in the npm package root (ie where it is), but I am happy with merging editorconfig / code style configuration with the extension. It will however have to be moved back later when we re-split. |
|
Thanks for the follow ups. Moving back to a separate repo is a good reason to keep things as is. |
For #300