Capture ios|android sourcemaps as deploy job artifacts #9641

kidroca · 2022-06-30T17:11:16Z

Details

Capture sourcemap files as deploy step artifacts
The artifacts can then be used in another job/workflow to decode Firebase Crashlytics stacktraces

Sample workflow

On Firebase Crashlytics event
Get stacktrace, platform and version from Crashlytics
Find the sourcemap artifacts for the given platform and version
Use npm run symbolicate:{platform} < crashlytics.stactrace.txt to decode the stack traces

The symbolicate command should be able to replace just the stack trace content of a text containing stack traces leaving the rest of the text intact

Storing and sharing artifacts between workflows: https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts

Fixed Issues

$ #9293

Tests

Requirements to be able to test:

Have JS stack traces (from Firebase Crashlytics)
know the platform, app version of the stack trace
find the commit which released the given app version
check out a new branch from it and cherry pick the changes from this merge request
- this is so we can (re)generate the same source and create source maps from it
- if this seems like to much work you can try to use this PR and try or force errors/stack traces for the current version

Test Steps

Verify production builds are producing source map files:
- iOS:
  1. run npx react-native run-ios --configuration Release --device "Your Device Name"
  2. Wait for the build to succeed
  3. A file called main.jsbundle.map should be available inside the project root - Expensify/App/main.jsbundle.map
- Android:
  1. run npx react-native run-android --variant=release
  2. Wait for the build to succeed
  3. A file android/app/build/generated/sourcemaps/react/release/index.android.bundle.map should be available
Use the generated source maps to symbolicate (decode) the stack traces
- use the new symbolicate:{platform} commands
- stacktraces are expected to come from stdin, for testing purposes we can reroute stdin to a file
  1. Prepare some stack traces to a {platform}.stacktrace.txt file
  2. Build a source map for the version of the app/platform these stack traces are from
  3. feed that file to the symbolicate command
```
npm run symbolicate:ios < ios.stacktrace.txt > ios.decoded.stacktrace.txt
```
  1. Decoded stack traces should be pointing to original files, some easy to distinguish examples are Onyx calls that point to node_modules/react-native-onyx/lib/Onyx.js

Notes on stack traces format

Decoding would fail or be incorrect if the stack traces format does not match
The format should be {anyText}:{lineNumber}:{colNumber}
anyText is preserved while lineNumber:colNumber are replaced with original file and original line:column
Line numbers should start from 1 (not 0)
More details and examples of valid/invalid and decoded stack traces can be seen here: #9293 (comment)

Example

Platform: Android
Version: 1.1.70-1

Minified stack trace content

.keyChanged (address at index.android.bundle:1:816124:)

Steps

Checkout version 1.1.70-1
Cherry pick commits from this PR
Build the Android app (to generate the .map file)
Save the above example stack trace as android.stacktrace.txt
Run npm run symbolicate:android < android.stacktrace.txt > android.decoded.stacktrace.txt

alternative usage: Instead of doing 4 and 5 is to use the symbolicate command per line and number like so
```
npm run symbolicate:android 1 816124
```

stack trace is decoded to

.keyChanged (address at node_modules/react-native-onyx/lib/Onyx.js:319:keyChanged:)

PR Review Checklist

Contributor (PR Author) Checklist

PR Reviewer Checklist

QA Steps

After this is deployed, verify that you can download the sourcemap artifacts for the platformDeploy.yml workflow run in which this was deployed.

No code changes
Minimal changes to ios build process - the app should still build and launch as before

Screenshots

Web

n/a

Mobile Web

n/a

Desktop

n/a

iOS

Android

"Bundle React Native code and images" is the phase that generates the JS bundle using `node_modules/react-native/scripts/react-native-xcode.sh` Providing a SOURCEMAP_FILE env variable would make the script generate a `.map` at build time The file can then be accessed from App project root as `ios.jsbundle.map`

At the moment there's a big in react-native with SOURCEMAP_FILE and hermes enabled facebook/react-native#32497

roryabraham · 2022-07-08T06:04:16Z

package.json

@@ -32,7 +32,9 @@
    "gh-actions-build": "./.github/scripts/buildActions.sh",
    "gh-actions-validate": "./.github/scripts/validateActionsAndWorkflows.sh",
    "analyze-packages": "ANALYZE_BUNDLE=true webpack --config config/webpack/webpack.common.js --env.envFile=.env.production",
-    "check-metro-bundler-port": "node config/checkMetroBundlerPort.js"
+    "check-metro-bundler-port": "node config/checkMetroBundlerPort.js",
+    "symbolicate:android": "npx metro-symbolicate android/app/build/generated/sourcemaps/react/release/index.android.bundle.map",


Are these scripts used by react native automatically?

Are these scripts used by react native automatically?

I'm not sure what do you mean - how would they be used automatically by react-native?

I've outlined sample usage in the "Details" section.

Something like this:

Build pipeline runs and saves sourcemaps somewhere

Firebase Crashlytics captures a crash

Something uses crash report data to retrieve the source maps for the build (version)

And runs npm run symbolicate:{platform} < crashlytics.stactrace.txt where .stacktrace.txt are js stack traces from "Firebase Crashlytics"

Awesome, thanks for the explanation. Sounds like a good plan

johnmlee101 · 2022-07-14T15:53:12Z

@roryabraham you mind being a full reviewer here as well? I'm a bit out of my understanding here and want to make sure I have a qualified set of second 👀

roryabraham · 2022-07-14T16:11:27Z

.github/workflows/platformDeploy.yml

@@ -93,6 +93,12 @@ jobs:
        env:
          VERSION: ${{ env.VERSION_CODE }}

+      - name: Archive Android sourcemaps
+        uses: actions/upload-artifact@v3


Just noting here that even though we normally use SHAs for community actions instead of vX tags, I think it might be fine to make exceptions for GitHub-maintained actions in the actions/ org.

What do you think @AndrewGable ?

roryabraham · 2022-07-14T16:32:56Z

package.json

@@ -32,7 +32,9 @@
    "gh-actions-build": "./.github/scripts/buildActions.sh",
    "gh-actions-validate": "./.github/scripts/validateActionsAndWorkflows.sh",
    "analyze-packages": "ANALYZE_BUNDLE=true webpack --config config/webpack/webpack.common.js --env.envFile=.env.production",
-    "check-metro-bundler-port": "node config/checkMetroBundlerPort.js"
+    "check-metro-bundler-port": "node config/checkMetroBundlerPort.js",
+    "symbolicate:android": "npx metro-symbolicate android/app/build/generated/sourcemaps/react/release/index.android.bundle.map",


Awesome, thanks for the explanation. Sounds like a good plan

melvin-bot · 2022-07-14T16:38:42Z

npm has a package.json file and a package-lock.json file. It seems you updated one without the other, which is usually a sign of a mistake. If you are updating a package make sure that you update the version in package.json then run npm install

melvin-bot · 2022-07-14T17:54:42Z

npm has a package.json file and a package-lock.json file. It seems you updated one without the other, which is usually a sign of a mistake. If you are updating a package make sure that you update the version in package.json then run npm install

kidroca · 2022-07-15T08:14:55Z

npm has a package.json file and a package-lock.json file. It seems you updated one without the other, which is usually a sign of a mistake. If you are updating a package make sure that you update the version in package.json then run npm install

The change is intentional - we added new scripts to package.json which doesn't change package-lock.json

johnmlee101

AFAICT it looks good on my end, will wait for the final comments/review

roryabraham · 2022-07-15T17:43:44Z

So I got an Android production build working and see the sourcemap being generated. So that's good 👍

However, using the following stacktrace copy/pasted from Firebase Crashlytics:

Non-fatal Exception: io.invertase.firebase.crashlytics.JavaScriptError: Cannot read property split of undefined
       at .getTextMatrix(address at index.android.bundle:1:3403705)
       at .WrappedText(address at index.android.bundle:1:3403814)
       at .renderWithHooks(address at index.android.bundle:1:157704)
       at .beginWork$1(address at index.android.bundle:1:197179)
       at .performUnitOfWork(address at index.android.bundle:1:183344)
       at .workLoopSync(address at index.android.bundle:1:183202)
       at .renderRootSync(address at index.android.bundle:1:183086)
       at .performSyncWorkOnRoot(address at index.android.bundle:1:180456)
       at .flushSyncCallbacks(address at index.android.bundle:1:147980)
       at .scheduleUpdateOnFiber(address at index.android.bundle:1:178108)
       at .enqueueSetState(address at index.android.bundle:1:192727)
       at .anonymous(address at index.android.bundle:1:107204)
       at .setWithOnyxState(address at index.android.bundle:1:983054)
       at .sendDataToConnection(address at index.android.bundle:1:818564)
       at .anonymous(address at index.android.bundle:1:819361)
       at .tryCallOne(InternalBytecode.js:53:16)
       at .anonymous(InternalBytecode.js:139:27)
       at .apply((native):0:0)
       at .anonymous(address at index.android.bundle:1:218386)
       at ._callTimer(address at index.android.bundle:1:217285)
       at ._callReactNativeMicrotasksPass(address at index.android.bundle:1:217489)
       at .callReactNativeMicrotasks(address at index.android.bundle:1:219482)
       at .__callReactNativeMicrotasks(address at index.android.bundle:1:124122)
       at .anonymous(address at index.android.bundle:1:123207)
       at .__guard(address at index.android.bundle:1:123963)
       at .flushedQueue(address at index.android.bundle:1:123118)
       at .callFunctionReturnFlushedQueue(address at index.android.bundle:1:122974)

Then running npm run symbolicate:android < stackTrace.txt > android.decoded.stacktrace.txt yielded an error:

TypeError: Line must be greater than or equal to 1, got 0

kidroca · 2022-07-16T05:31:03Z

Then running npm run symbolicate:android < stackTrace.txt > android.decoded.stacktrace.txt yielded an error:

TypeError: Line must be greater than or equal to 1, got 0

I had the same error, I've updated the description with

Line numbers should start from 1 (not 0)
More details and examples of valid/invalid and decoded stack traces can be seen here: #9293 (comment)

The line at .apply((native):0:0) should be removed
Lines like these should be filtered out

roryabraham · 2022-07-18T21:53:50Z

Cool, that worked 👍

My only other critique is that when we execute this via npm script i.e: npm run android:symbolicate, then we get some default npm stuff in the output that we don't want:

> [email protected] symbolicate:android /Users/roryabraham/Expensidev/App
> npx metro-symbolicate android/app/build/generated/sourcemaps/react/release/index.android.bundle.map

We can workaround that by just adding the --silent flag npm run --silent android:symbolicate < stackTrace.txt > android.decoded.stacktrace.txt, so NAB

Just two things to take note of that we'll need to workaround in the next phase where we hook up decoded stacktraces with the PHP <-> Firebase integration (cc @AndrewGable):

Need to remove any lines from the stacktrace that include any zero line numbers. Maybe a regex like this would do:
```
```
/^.*:(?:0:.*|.*:0)\)$/gm
```
```
Also want to run the npm run symbolicate:{platform} script with the --silent tag

melvin-bot · 2022-07-18T21:54:25Z

Triggered auto assignment to @PauloGasparSv (InternalQA), see https://stackoverflow.com/c/expensify/questions/5042 for more details.

melvin-bot · 2022-07-18T21:54:28Z

@roryabraham looks like this was merged without passing tests. Please add a note explaining why this was done and remove the Emergency label if this is not an emergency.

roryabraham · 2022-07-18T21:54:45Z

Not an emergency, tests were passing

OSBotify · 2022-07-18T22:00:41Z

✋ This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

kidroca · 2022-07-19T16:47:57Z

@roryabraham
Another option to filter unwanted shell output might be to define the stuff in ./scripts and not have it in package.json
Or (depending on how the source map file is retrieved) directly run npx metro-symbolicate map-file.map < stacks.txt > result.txt

roryabraham · 2022-07-19T21:38:01Z

Checking this off for internal QA since it was a pass:

OSBotify · 2022-07-25T17:21:11Z

🚀 Deployed to production by @luacmartins in version: 1.1.85-8 🚀

platform	result
🤖 android 🤖	success ✅
🖥 desktop 🖥	success ✅
🍎 iOS 🍎	success ✅
🕸 web 🕸	success ✅

kidroca added 4 commits June 30, 2022 16:47

Add symbolicate:(ios|android) package scripts

ac455fe

Expose ios/android sourcemaps as artifacts from platformDeploy.yml

db7bc3a

Use EXTRA_PACKAGER_ARGS instead of SOURCEMAP_FILE

eabc197

At the moment there's a big in react-native with SOURCEMAP_FILE and hermes enabled facebook/react-native#32497

kidroca force-pushed the kidroca/feature/sourcemaps-alt branch from d6378f2 to eabc197 Compare June 30, 2022 17:13

kidroca mentioned this pull request Jun 30, 2022

Generate source maps for iOS and Android #9596

Closed

91 tasks

roryabraham reviewed Jul 8, 2022

View reviewed changes

kidroca mentioned this pull request Jul 11, 2022

Include source maps for staging apps #9293

Closed

kidroca marked this pull request as ready for review July 14, 2022 13:15

kidroca requested a review from a team as a code owner July 14, 2022 13:15

melvin-bot bot requested review from johnmlee101 and removed request for a team July 14, 2022 13:15

roryabraham reviewed Jul 14, 2022

View reviewed changes

roryabraham assigned kidroca Jul 14, 2022

roryabraham added the InternalQA This pull request required internal QA label Jul 14, 2022

johnmlee101 approved these changes Jul 15, 2022

View reviewed changes

roryabraham approved these changes Jul 18, 2022

View reviewed changes

roryabraham merged commit 4f27bd1 into Expensify:main Jul 18, 2022

melvin-bot bot assigned PauloGasparSv Jul 18, 2022

melvin-bot bot added the Emergency label Jul 18, 2022

roryabraham removed the Emergency label Jul 18, 2022

OSBotify mentioned this pull request Jul 18, 2022

Deploy Checklist: New Expensify 2022-07-18 #9970

Closed

79 tasks

kidroca deleted the kidroca/feature/sourcemaps-alt branch July 19, 2022 16:48

eVoloshchak mentioned this pull request Oct 4, 2022

[$250] Investigate: 💥 Crash 💥 value #11554

Closed

hannojg mentioned this pull request Jun 17, 2024

[HOLD for payment 2024-07-10] [HOLD for payment 2024-07-02] [HOLD for payment 2024-06-28] [HOLD for payment 2024-06-28] MEDIUM: Enable Function Symbolication for Detailed Performance Tracing #43649

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Capture ios|android sourcemaps as deploy job artifacts #9641

Capture ios|android sourcemaps as deploy job artifacts #9641

kidroca commented Jun 30, 2022 •

edited

Loading

roryabraham Jul 8, 2022

kidroca Jul 11, 2022

roryabraham Jul 14, 2022

johnmlee101 commented Jul 14, 2022

roryabraham Jul 14, 2022

roryabraham Jul 14, 2022

melvin-bot bot commented Jul 14, 2022

melvin-bot bot commented Jul 14, 2022

kidroca commented Jul 15, 2022

johnmlee101 left a comment

roryabraham commented Jul 15, 2022

kidroca commented Jul 16, 2022

roryabraham commented Jul 18, 2022 •

edited

Loading

melvin-bot bot commented Jul 18, 2022

melvin-bot bot commented Jul 18, 2022

roryabraham commented Jul 18, 2022

OSBotify commented Jul 18, 2022

kidroca commented Jul 19, 2022

roryabraham commented Jul 19, 2022

OSBotify commented Jul 25, 2022

Capture ios|android sourcemaps as deploy job artifacts #9641

Capture ios|android sourcemaps as deploy job artifacts #9641

Conversation

kidroca commented Jun 30, 2022 • edited Loading

Details

Fixed Issues

Tests

Requirements to be able to test:

Test Steps

Notes on stack traces format

Example

Minified stack trace content

Steps

stack trace is decoded to

PR Review Checklist

Contributor (PR Author) Checklist

PR Reviewer Checklist

QA Steps

Screenshots

Web

Mobile Web

Desktop

iOS

Android

roryabraham Jul 8, 2022

Choose a reason for hiding this comment

kidroca Jul 11, 2022

Choose a reason for hiding this comment

roryabraham Jul 14, 2022

Choose a reason for hiding this comment

johnmlee101 commented Jul 14, 2022

roryabraham Jul 14, 2022

Choose a reason for hiding this comment

roryabraham Jul 14, 2022

Choose a reason for hiding this comment

melvin-bot bot commented Jul 14, 2022

melvin-bot bot commented Jul 14, 2022

kidroca commented Jul 15, 2022

johnmlee101 left a comment

Choose a reason for hiding this comment

roryabraham commented Jul 15, 2022

kidroca commented Jul 16, 2022

roryabraham commented Jul 18, 2022 • edited Loading

melvin-bot bot commented Jul 18, 2022

melvin-bot bot commented Jul 18, 2022

roryabraham commented Jul 18, 2022

OSBotify commented Jul 18, 2022

kidroca commented Jul 19, 2022

roryabraham commented Jul 19, 2022

OSBotify commented Jul 25, 2022

kidroca commented Jun 30, 2022 •

edited

Loading

roryabraham commented Jul 18, 2022 •

edited

Loading