Add more simulators in Xcode 15.0

[amseddi]

Tool name

Xcode 15.0

Tool license

NA

Add or update?

• Add
• Update

Desired version

16.4 Simulators

Approximate size

No response

Brief description of tool

Add more simulators for Xcode 15.0 or at least iOS 16.4

URL for tool's homepage

No response

Provide a basic test case to validate the tool's functionality.

No response

Platforms where you need the tool

• Azure DevOps
• GitHub Actions

Runner images where you need the tool

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• Windows Server 2019
• Windows Server 2022

Can this tool be installed during the build?

No response

Tool installation time in runtime

No response

Are you willing to submit a PR?

No response

[ilia-shipitsin]

I created small repro

          brew install xcodesorg/made/xcodes
          xcodes runtimes --include-betas

https://github.com/ilia-shipitsin/xcodes15-runtimes/actions/runs/6337845954/job/17213741033

iOS 16.4 is already there

[ilia-shipitsin]

[amseddi]

I am referring to this:
https://github.com/actions/runner-images/blob/main/images/macos/macos-13-Readme.md#installed-simulators

As you can see, only iOS 17.0 simulator is available in Xcode 15.0

[amseddi]

I created small repro

          brew install xcodesorg/made/xcodes
          xcodes runtimes --include-betas

https://github.com/ilia-shipitsin/xcodes15-runtimes/actions/runs/6337845954/job/17213741033

iOS 16.4 is already there

In this build you are using Xcode 14.3.1 as it is the default and it indeed contains iOS 17.0 and 16.4 simulators
(https://github.com/actions/runner-images/blob/main/images/macos/macos-13-Readme.md#xcode)

[amseddi]

FYI This would help speeding some build times as there is a known issue with iOS 17 simulators performance

https://developer.apple.com/documentation/xcode-release-notes/xcode-15-release-notes#Known-Issues

Executing Unit/UI tests from Xcode on the iOS Simulator takes an extended time to launch on first run of the suite. (115187363) (110330776) (FB12237092)

[ilia-shipitsin]

we investigate simulators

#8023
#7971

thanks for the tip, performance of simulators is currently poor

[ilia-shipitsin]

ntains iOS 17.0 and 16.4 simulator

simulators are system wide as far as I know.

I'll check with xcode-select a little bit later

[amseddi]

Not necessarily: some are system wide and some are bundled with Xcode.

I forked your sample repo and selected Xcode 15.0 (https://github.com/amseddi/xcodes15-runtimes)

And this is the result:
https://github.com/amseddi/xcodes15-runtimes/actions/runs/6338503994/job/17215753615

[ilia-shipitsin]

interesting. let me check it

(we install runtimes in a loop, but we do not run xcode-select before adding. I wonder if that is required)

[ilia-shipitsin]

I tried to install on clean vm.

I installed XCode-14.3.1, XCode-15.0.0, I did "xcode-select" and then "xcodebuild -downloadAllPlatforms".

even running "xcode-select" did not change anything. I see iOS-16.4 is available for XCode-14.3.1 and iOS-17.0 for XCode-15.0.0

what is your local experience ? what do you see running "xcodes runtimes" ?
if you have it working locally and you can tell us how to install simulators, we can try

[ilia-shipitsin]

for example

Mac-mini:~ runner$ sudo xcode-select -s /Applications/Xcode-15.0.0.app/
Mac-mini:~ runner$ xcodebuild  -downloadPlatform 'iOS 16.4'
Could not find platform: iOS 16.4
Mac-mini:~ runner$

[ilia-shipitsin]

also, I do not see an evidence that XCode 15 supports "iOS 16.4" officially

https://developer.apple.com/documentation/xcode-release-notes/xcode-15-release-notes

please elaborate with "iOS 16.4" support on XCode 15. if you have it working on your local Mac, please share details how did you install it

[amseddi]

I usually add it using the Xcode user interface.

• Go to Settings > Platforms

• Click on + > iOS... and then select and download iOS 16.4

The Apple documentation explains how to do it from the command line but you need to download the dmg first from the Apple developer website.

[amseddi]

And reading the xcodes docs, the tool can install runtimes too:

xcodes runtimes install "iOS 16.4"

[ilia-shipitsin]

[ilia-shipitsin]

@amseddi , I suggest you to try installing it within your pipeline (yes, it takes time). as for installing - it looks like it can be installed that way. why I'm asking to try - I'm not sure whether it will work.

if that works and you confirm that, we can install it

[amseddi]

I could add it successfully to my workflow

And I am seeing better tests execution times than with the 17.0 simulator.

[amseddi]

We could add it to the workflow cache but IMO it is better to add it to the runner so everyone can profit from this.

[ilia-shipitsin]

no, I do not mean cache as long-term approach, it's just for testing. Since you confirm it works for you, thanks, we'll discuss internally adding runtimes.

[AlexCookeTrader]

I'm also having the same issue where build times are identical across 14.2 and 15, yet test runs on iOS17 simulators have taken us over our 1 hour limit for individual jobs. Hoping for an update with iOS 6.4 support soon.

[ilia-shipitsin]

action https://github.com/muukii/actions-xcode-install-simulator is good indeed.
maybe, we can leave images small and required runtimes can be installed (and cached) by that action

[amseddi]

I am afraid it is not a viable solution: I tested the action and getting the cache took the same time as downloading the simulator runtime.

I would prefer to have it included in the image instead of adding 7 min to each build.

[amseddi]

Maybe you could have a rule of adding the previous runtime with each new version of Xcode 🤔

[joshuapoq]

If this may take a while could we look to get the Xcode 15.1 beta added to the runners?

Fixed: Executing Unit/UI tests from Xcode on the iOS Simulator takes an extended time to launch on first run of the suite. (115187363) (110330776)

[eseay]

Bumping this issue. My specific use case is:

• We are compiling our apps/packages with Xcode 15.x
• Need to execute unit tests against iOS 16.4 (at least - ideally also iOS 15)
• Installing these simulators at runtime is a non-starter because of the performance and time cost.

[AlexCookeTrader]

Bumping this issue as it is currently blocking our team. Hoping that the simulator performance issues are being addressed.

[amseddi]

@ilia-shipitsin Do you have any news on this issue?

[einar-notland]

This might be of help. We currently use this step in our CI workflow to run tests on 16.4 simulator. The step creates a simulator called "iPhone 13" so we can reference it under devices in our fastlane scan.

- name: Link to 16.4 Simulators
        run: |
          echo "Creating Runtimes folder if needed..."
          sudo mkdir -p /Library/Developer/CoreSimulator/Profiles/Runtimes
          echo "Creating symlink of the iOS 16.4 runtime..."
          sudo ln -s /Applications/Xcode_14.3.1.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime /Library/Developer/CoreSimulator/Profiles/Runtimes/iOS\ 16.4.simruntime
          xcrun simctl create "iPhone 13" "com.apple.CoreSimulator.SimDeviceType.iPhone-13" "com.apple.CoreSimulator.SimRuntime.iOS-16-4"

[AlexCookeTrader]

This might be of help. We currently use this step in our CI workflow to run tests on 16.4 simulator. The step creates a simulator called "iPhone 13" so we can reference it under devices in our fastlane scan.

- name: Link to 16.4 Simulators
        run: |
          echo "Creating Runtimes folder if needed..."
          sudo mkdir -p /Library/Developer/CoreSimulator/Profiles/Runtimes
          echo "Creating symlink of the iOS 16.4 runtime..."
          sudo ln -s /Applications/Xcode_14.3.1.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime /Library/Developer/CoreSimulator/Profiles/Runtimes/iOS\ 16.4.simruntime
          xcrun simctl create "iPhone 13" "com.apple.CoreSimulator.SimDeviceType.iPhone-13" "com.apple.CoreSimulator.SimRuntime.iOS-16-4"

I'm seeing some good results with this! However it would seem the coverage results are not being generated or saved in the usual spot using this method. Going to keep digging for now unless anyone has encountered and resolved this issue.

[mikhailkoliada]

Hello and sorry for long delay! We do not plan providing additional simulators (other than the ones which come in stock with the release) as a part of the macOS runners, because we can not guarantee it will work well (or if it will work at all), please use additional 3rd party tools to install the versions you need in runtime.