This repository has been archived by the owner on Jun 1, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 204
Install shopify-extensions during gem installation #1496
Merged
t6d
merged 6 commits into
main
from
feature/download-shopify-cli-extensions-during-installation
Sep 17, 2021
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
f7c5190
Install extension server
t6d abd7361
Rake tasks for managing shopify-extensions
t6d ddfd179
Remove API calls to Github
t6d e3890eb
Remove unnecessary dependencies
t6d 0f98a82
rake extensions:install & disable auto install
t6d 646eb64
shopify-extensions version
t6d File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
require_relative "./shopify_extensions" | ||
|
||
File.write("Makefile", <<~MAKEFILE) | ||
.PHONY: clean | ||
|
||
clean: ; | ||
|
||
install: ; | ||
MAKEFILE | ||
|
||
begin | ||
ShopifyExtensions.install( | ||
target: File.join(File.dirname(__FILE__), "shopify-extensions") | ||
) | ||
rescue ShopifyExtensions::InstallationError => error | ||
STDERR.puts(error.message) | ||
rescue => error | ||
STDERR.puts("Unable to install shopify-extensions: #{error}") | ||
end | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The rescue statements allow the installation of the binary to fail silently. This should eventually be changed to fail the installation and notify the user to file a bug report. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,152 @@ | ||
require "rbconfig" | ||
require "open-uri" | ||
require "zlib" | ||
require "open3" | ||
|
||
module ShopifyExtensions | ||
def self.install(**args) | ||
Install.call(**args) | ||
end | ||
|
||
class Install | ||
def self.call(platform: Platform.new, **args) | ||
new.call(platform: platform, **args) | ||
end | ||
|
||
def self.version | ||
File.read(File.expand_path("../version", __FILE__)).strip | ||
end | ||
|
||
def call(platform:, version: self.class.version, target:) | ||
target = platform.format_path(target.to_s) | ||
|
||
asset = Asset.new( | ||
platform: platform, | ||
version: version, | ||
owner: "Shopify", | ||
repository: "shopify-cli-extensions", | ||
basename: "shopify-extensions" | ||
) | ||
downloaded = asset.download(target: target) | ||
raise InstallationError.asset_not_found(platform: platform, version: version) unless downloaded | ||
|
||
raise InstallationError.installation_failed unless verify(target, version: version) | ||
end | ||
|
||
private | ||
|
||
def fetch_release_details_for(version:) | ||
JSON.parse(URI.parse(release_url_for(version: version)).open.read).yield_self(&Release) | ||
rescue OpenURI::HTTPError | ||
nil | ||
end | ||
|
||
def verify(target, version:) | ||
return false unless File.executable?(target) | ||
installed_server_version, exit_code = Open3.capture2(target, "version") | ||
return false unless exit_code == 0 | ||
return false unless installed_server_version.strip == version.strip | ||
true | ||
end | ||
end | ||
|
||
class InstallationError < RuntimeError | ||
def self.installation_failed | ||
new("Failed to install shopify-extensions properly") | ||
end | ||
|
||
def self.asset_not_found(platform:, version:) | ||
new(format( | ||
"Unable to download shopify-extensions %{version} for %{os} (%{cpu})", | ||
version: version, | ||
os: platform.os, | ||
cpu: platform.cpu | ||
)) | ||
end | ||
end | ||
|
||
Asset = Struct.new(:platform, :version, :owner, :repository, :basename, keyword_init: true) do | ||
def download(target:) | ||
Dir.chdir(File.dirname(target)) do | ||
File.open(File.basename(target), "wb") do |target_file| | ||
decompress(url.open, target_file) | ||
end | ||
File.chmod(0755, target) | ||
end | ||
|
||
true | ||
rescue OpenURI::HTTPError | ||
false | ||
end | ||
|
||
def url | ||
URI.parse(format( | ||
"https://github.com/%{owner}/%{repository}/releases/download/%{version}/%{filename}", | ||
owner: owner, | ||
repository: repository, | ||
version: version, | ||
filename: filename | ||
)) | ||
end | ||
|
||
def filename | ||
format( | ||
"%{basename}-%{os}-%{cpu}.%{extension}", | ||
basename: basename, | ||
os: platform.os, | ||
cpu: platform.cpu, | ||
extension: platform.os == "windows" ? "exe.gz" : "gz" | ||
) | ||
end | ||
|
||
private | ||
|
||
def decompress(source, target) | ||
zlib = Zlib::GzipReader.new(source) | ||
target << zlib.read | ||
ensure | ||
zlib.close | ||
end | ||
end | ||
|
||
Platform = Struct.new(:ruby_config) do | ||
def initialize(ruby_config = RbConfig::CONFIG) | ||
super(ruby_config) | ||
end | ||
|
||
def format_path(path) | ||
case os | ||
when "windows" | ||
File.extname(path) != ".exe" ? path + ".exe" : path | ||
else | ||
path | ||
end | ||
end | ||
|
||
def to_s | ||
format("%{os}-%{cpu}", os: os, cpu: cpu) | ||
end | ||
|
||
def os | ||
case ruby_config.fetch("host_os") | ||
when /linux/ | ||
"linux" | ||
when /darwin/ | ||
"darwin" | ||
else | ||
"windows" | ||
end | ||
end | ||
|
||
def cpu | ||
case ruby_config.fetch("host_cpu") | ||
when /arm.*64/ | ||
"arm64" | ||
when /64/ | ||
"amd64" | ||
else | ||
"386" | ||
end | ||
end | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
v0.1.0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,153 @@ | ||
require "test_helper" | ||
require_relative "../../../ext/shopify-extensions/shopify_extensions.rb" | ||
|
||
module ShopifyExtensions | ||
class ShopfyExtensionsTest < Minitest::Test | ||
def test_installation_of_existing_version_for_mac_os | ||
stub_executable_download | ||
|
||
target = File.join(Dir.mktmpdir, "shopify-extensions") | ||
|
||
Install.call( | ||
platform: Platform.new({ | ||
"host_os" => "darwin20.3.0", | ||
"host_cpu" => "x86_64", | ||
}), | ||
version: "v0.1.0", | ||
target: target | ||
) | ||
|
||
assert File.file?(target) | ||
assert File.executable?(target) | ||
assert_match(/v0.1.0/, %x(#{target})) | ||
end | ||
|
||
def test_installation_of_existing_version_for_windows | ||
stub_executable_download | ||
|
||
target = File.join(Dir.mktmpdir, "shopify-extensions.exe") | ||
|
||
Install.call( | ||
platform: Platform.new({ | ||
"host_os" => "mingw32", | ||
"host_cpu" => "x64", | ||
}), | ||
version: "v0.1.0", | ||
target: target | ||
) | ||
|
||
assert File.file?(target) | ||
assert File.executable?(target) | ||
end | ||
|
||
def test_handle_http_errors_during_asset_download | ||
simulate_broken_asset_link | ||
|
||
target = File.join(Dir.mktmpdir, "shopify-extensions") | ||
|
||
assert_raises(InstallationError) do | ||
Install.call( | ||
platform: Platform.new({ | ||
"host_os" => "darwin20.3.0", | ||
"host_cpu" => "x86_64", | ||
}), | ||
version: "v0.1.0", | ||
target: target | ||
) | ||
end | ||
end | ||
|
||
def test_incorrect_binary | ||
stub_executable_download | ||
|
||
target = File.join(Dir.mktmpdir, "shopify-extensions.exe") | ||
File.expects(:executable?).with(target).returns(false) | ||
|
||
error = assert_raises(InstallationError) do | ||
Install.call( | ||
platform: Platform.new({ | ||
"host_os" => "mingw32", | ||
"host_cpu" => "x64", | ||
}), | ||
version: "v0.1.0", | ||
target: target | ||
) | ||
end | ||
|
||
assert_equal "Failed to install shopify-extensions properly", error.message | ||
end | ||
|
||
class PlatformTest < MiniTest::Test | ||
def test_recognizes_linux | ||
linux_vm = ruby_config(os: "linux-gnu", cpu: "x86_64") | ||
assert_equal "linux-amd64", Platform.new(linux_vm).to_s | ||
end | ||
|
||
def test_recognices_mac_os | ||
t6d marked this conversation as resolved.
Show resolved
Hide resolved
|
||
intel_mac = ruby_config(os: "darwin20.3.0", cpu: "x86_64") | ||
m1_mac = ruby_config(os: "darwin20.3.0", cpu: "arm64") | ||
|
||
assert_equal "darwin-amd64", Platform.new(intel_mac).to_s | ||
assert_equal "darwin-arm64", Platform.new(m1_mac).to_s | ||
end | ||
|
||
def test_recognices_windows | ||
t6d marked this conversation as resolved.
Show resolved
Hide resolved
|
||
windows_vm_64_bit = ruby_config(os: "mingw32", cpu: "x64") | ||
windows_vm_32_bit = ruby_config(os: "mingw32", cpu: "i686") | ||
assert_equal "windows-amd64", Platform.new(windows_vm_64_bit).to_s | ||
assert_equal "windows-386", Platform.new(windows_vm_32_bit).to_s | ||
end | ||
|
||
def test_adds_exe_extension_to_binaries_on_windows | ||
windows_vm_64_bit = ruby_config(os: "mingw32", cpu: "x64") | ||
assert_equal "some/command.exe", Platform.new(windows_vm_64_bit).format_path("some/command") | ||
end | ||
|
||
private | ||
|
||
def ruby_config(os:, cpu:) | ||
{ | ||
"host_os" => os, | ||
"host_cpu" => cpu, | ||
} | ||
end | ||
end | ||
|
||
def stub_executable_download | ||
dummy_archive = load_dummy_archive | ||
|
||
stub_request(:get, "https://github.com/Shopify/shopify-cli-extensions/releases/download/v0.1.0/shopify-extensions-windows-amd64.exe.gz") | ||
.to_return( | ||
status: 200, | ||
headers: { | ||
"Content-Type" => "application/octet-stream", | ||
"Content-Disposition" => "attachment; filename=shopify-extensions-windows-amd64.exe.gz", | ||
"Content-Length" => dummy_archive.size, | ||
}, | ||
body: dummy_archive | ||
) | ||
|
||
stub_request(:get, "https://github.com/Shopify/shopify-cli-extensions/releases/download/v0.1.0/shopify-extensions-darwin-amd64.gz") | ||
.to_return( | ||
status: 200, | ||
headers: { | ||
"Content-Type" => "application/octet-stream", | ||
"Content-Disposition" => "attachment; filename=shopify-extensions-darwin-amd64.gz", | ||
"Content-Length" => dummy_archive.size, | ||
}, | ||
body: dummy_archive | ||
) | ||
end | ||
|
||
def simulate_broken_asset_link | ||
stub_request(:get, "https://github.com/Shopify/shopify-cli-extensions/releases/download/v0.1.0/shopify-extensions-darwin-amd64.gz") | ||
.to_raise(OpenURI::HTTPError.new("404 Not Found", StringIO.new)) | ||
end | ||
|
||
def load_dummy_archive | ||
path = File.expand_path("../../../fixtures/shopify-extensions.gz", __FILE__) | ||
raise "Dummy archive not found: #{path}" unless File.file?(path) | ||
File.read(path) | ||
end | ||
end | ||
end |
Binary file not shown.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I might have missed the discussion, but what's the motivation behind pulling the binary at installation time as opposed to vendoring it as part of releasing the package? Is it because we don't want to include the binaries of architectures other than ours?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have we considered Nokogiri's approach where multiple native gems are generated containing the right binary for each architecture?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That is correct. We've considered this option but dismissed it due to binary size. The binaries are quite largue as each one includes the Go runtime: ~ 10 MB per binary (5 MB compressed)
We have not. I wasn't aware that this option exists. From the link you provided, it's not immediately evident to me how to actually package and publish OS specific versions of Ruby gems. Do you have any experience with this yourself and could elaborate a bit?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The approach we took follows the one that ESBuild uses to obtain its binary when you install the corresponding node package. Can you expand a bit on the concerns you have with this approach?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pepibumur, do you see this as a blocker or something that we could improve incrementally. Since we're not releasing to partners before my intermission, I'd be ok to wait with integrating that, but it makes testing this a little more manual.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After some discussion, we decided that this can be an incremental improvement and is no reason to hold back on this PR. Especially now that the installation of
shopify-extensions
isn't done during gem installation for now. I'll investigate whether I want to pivot the approach when I'm back from intermission.That said, now that we're no longer making API calls to GitHub but directly construct the download URL, I'm inclined to stick to the current approach.