swarm travis

[thedavidmeister]

changes

• turns travis into a swarm of builds
• turns on thinlto and incremental builds (i think it works?)
  • got the flags from Tracking issue for enabling multiple CGUs in release mode by default rust-lang/rust#45320 (comment)
• utterly thrashes total build time up 2.5 hours
• brings wall time down to ~30 mins cold and 17.5 mins warm
• introduces a cfg( feature = "broken-tests") flag so we can track and repair slow/flaky/broken tests in a structured way without just commenting them straight out
  • replaced some not(windows) attributes with this instead

obviously this only really works while we have many builds

these changes essentially make it so we can take advantage of the 13 builds we have and keep them busy adding value while we wait for upstream fixes

reason for not appveyor

• absolute max builds in cloud = 4, but only 1 on free account, even "pro" is only 2 builds
• cache limit is shared across all builds and is 1GB up to 5GB
• cache is fast but...
  • incremental builds make avoiding recompilation reliable but blow up cache limit
  • debug builds are too big
  • --release builds are smaller but don't often recompile
  • saw odd behaviour with unreliable caching
  • silent fails and bugs make caching unpredictable
• compilation is sloooow, windows in general seems to be slow for compilation
• 2 cpus only
• overall we end up right back at 40-50 minutes for useful work :(

followups

• installing rust-docs is now the slowest thing we do by far :( this is going back and forward upstream
  • Add a --minimal flag to rustup-init rust-lang/rustup#907
• bring thinglto and incremental builds across to circle
• general cleanup of CI across the board

---

notes duing testing

pros

• logs work, keeps most recent 2000 LOC https://ci.appveyor.com/project/thedavidmeister/holochain-rust/builds/21081130
• seems faster than travis
• seems to be what pretty much everyone in rust is using for windows, so can "borrow" config from elsewhere
• caching seems to work with recompilation/downloads
• looks nicer than travis (imo)
• has an on-premesis enterprise license we could use with our own metal

cons

• only one build at a time on free account, paid only has 2 or 4 builds - https://www.appveyor.com/docs/parallel-testing/#requirements
• cache dir has size limits linked to how much $$ paid for account, the size limit is small (~1GB) https://www.appveyor.com/docs/build-cache/#cache-size-beta
• each (free) machine only has 2 CPU and 4-7GB RAM https://www.appveyor.com/docs/build-environment/#build-vm-configurations
  • 2 CPU feels underpowered for compilation, i think this is the same problem we had on travis
• hard limit on build times at 60 minutes, which is more problematic because of limited parallel options
• cache is not immutable and is fragile
  • not as fragile as travis
  • pretty fragile, e.g. all caches dropped whenever config changes for any cache
  • i think all builds share same cache allowance in pricing
  • i think i found a bug, where extra whitespace in the yaml can cause caches to build but not restore :(
  • seems to be some kind of delay or something between producing a cache and having it show up in later builds
    • https://ci.appveyor.com/project/thedavidmeister/holochain-rust/builds/21105281 has cache for cas
    • https://ci.appveyor.com/project/thedavidmeister/holochain-rust/builds/21105472 next build does not see cas cache
    • https://ci.appveyor.com/project/thedavidmeister/holochain-rust/builds/21105534 does see it

notes

• appveyor is going nowhere

  • promising at first
  • 1GB shared limit across all builds rules out incremental cache i think
  • singular build means can't "swarm"

• downloading travis-docs is the worst! it adds 8 minutes to install and downloads 100's of mb

• thinlto is much better than --release

  • 390s cold vs 235 on travis
    • https://travis-ci.com/holochain/holochain-rust/jobs/166066228
    • https://travis-ci.com/holochain/holochain-rust/jobs/166070843

• cold 30 mins on wasm:

  • https://travis-ci.com/holochain/holochain-rust/jobs/166086949#L739

• really want this Add a --minimal flag to rustup-init rust-lang/rustup#907!! will drop times across the board by 5+ minutes

[lucksus]

Great!
Is feature "broken-tests" a thing? I can't find a custom definition for that...

[zippy]

Woot!!

[thedavidmeister]

@lucksus iirc https://doc.rust-lang.org/1.5.0/book/conditional-compilation.html means we can send --cfg feature="broken-tests" to rustc