Skip to content
This repository has been archived by the owner on Jun 1, 2023. It is now read-only.

Install shopify-extensions during gem installation #1496

Merged
merged 6 commits into from
Sep 17, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,4 @@ packaging/debian/shopify-cli.deb
packaging/rpm/build
packaging/rpm/shopify-cli.spec
.byebug_history
ext/shopify-extensions/shopify-extensions
ext/shopify-extensions/shopify-extensions
38 changes: 38 additions & 0 deletions Rakefile
Original file line number Diff line number Diff line change
Expand Up @@ -111,3 +111,41 @@ end

desc("Builds all distribution packages of the CLI")
task(package: "package:all")

namespace :extensions do
task :update do
version = ENV.fetch("VERSION").strip
error("Invalid version") unless /^v\d+\.\d+\.\d+/.match(version)
File.write(Paths.extension("version"), version)
end

task :symlink do
source = Paths.root("..", "shopify-cli-extensions", "shopify-extensions")
error("Unable to find shopify-extensions executable: #{executable}") unless File.executable?(source)
target = Paths.extension("shopify-extensions")
File.delete(target) if File.exist?(target)
File.symlink(source, target)
end

task :install do
target = Paths.extension("shopify-extensions")
require_relative Paths.extension("shopify_extensions.rb")
File.delete(target) if File.exist?(target)
ShopifyExtensions.install(target: target)
end

module Paths
def self.extension(*args)
root("ext", "shopify-extensions", *args)
end

def self.root(*args)
Pathname(File.dirname(__FILE__)).join(*args).to_s
end
end
end

def error(message, output: STDERR, code: 1)
output.puts(message)
exit(code)
end
19 changes: 19 additions & 0 deletions ext/shopify-extensions/extconf.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
require_relative "./shopify_extensions"

File.write("Makefile", <<~MAKEFILE)
.PHONY: clean

clean: ;

install: ;
MAKEFILE

begin
ShopifyExtensions.install(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might have missed the discussion, but what's the motivation behind pulling the binary at installation time as opposed to vendoring it as part of releasing the package? Is it because we don't want to include the binaries of architectures other than ours?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have we considered Nokogiri's approach where multiple native gems are generated containing the right binary for each architecture?

Copy link
Contributor Author

@t6d t6d Sep 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it because we don't want to include the binaries of architectures other than ours?

That is correct. We've considered this option but dismissed it due to binary size. The binaries are quite largue as each one includes the Go runtime: ~ 10 MB per binary (5 MB compressed)

Have we considered Nokogiri's approach where multiple native gems are generated containing the right binary for each architecture?

We have not. I wasn't aware that this option exists. From the link you provided, it's not immediately evident to me how to actually package and publish OS specific versions of Ruby gems. Do you have any experience with this yourself and could elaborate a bit?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The approach we took follows the one that ESBuild uses to obtain its binary when you install the corresponding node package. Can you expand a bit on the concerns you have with this approach?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pepibumur, do you see this as a blocker or something that we could improve incrementally. Since we're not releasing to partners before my intermission, I'd be ok to wait with integrating that, but it makes testing this a little more manual.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After some discussion, we decided that this can be an incremental improvement and is no reason to hold back on this PR. Especially now that the installation of shopify-extensions isn't done during gem installation for now. I'll investigate whether I want to pivot the approach when I'm back from intermission.

That said, now that we're no longer making API calls to GitHub but directly construct the download URL, I'm inclined to stick to the current approach.

target: File.join(File.dirname(__FILE__), "shopify-extensions")
)
rescue ShopifyExtensions::InstallationError => error
STDERR.puts(error.message)
rescue => error
STDERR.puts("Unable to install shopify-extensions: #{error}")
end
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rescue statements allow the installation of the binary to fail silently. This should eventually be changed to fail the installation and notify the user to file a bug report.

152 changes: 152 additions & 0 deletions ext/shopify-extensions/shopify_extensions.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
require "rbconfig"
require "open-uri"
require "zlib"
require "open3"

module ShopifyExtensions
def self.install(**args)
Install.call(**args)
end

class Install
def self.call(platform: Platform.new, **args)
new.call(platform: platform, **args)
end

def self.version
File.read(File.expand_path("../version", __FILE__)).strip
end

def call(platform:, version: self.class.version, target:)
target = platform.format_path(target.to_s)

asset = Asset.new(
platform: platform,
version: version,
owner: "Shopify",
repository: "shopify-cli-extensions",
basename: "shopify-extensions"
)
downloaded = asset.download(target: target)
raise InstallationError.asset_not_found(platform: platform, version: version) unless downloaded

raise InstallationError.installation_failed unless verify(target, version: version)
end

private

def fetch_release_details_for(version:)
JSON.parse(URI.parse(release_url_for(version: version)).open.read).yield_self(&Release)
rescue OpenURI::HTTPError
nil
end

def verify(target, version:)
return false unless File.executable?(target)
installed_server_version, exit_code = Open3.capture2(target, "version")
return false unless exit_code == 0
return false unless installed_server_version.strip == version.strip
true
end
end

class InstallationError < RuntimeError
def self.installation_failed
new("Failed to install shopify-extensions properly")
end

def self.asset_not_found(platform:, version:)
new(format(
"Unable to download shopify-extensions %{version} for %{os} (%{cpu})",
version: version,
os: platform.os,
cpu: platform.cpu
))
end
end

Asset = Struct.new(:platform, :version, :owner, :repository, :basename, keyword_init: true) do
def download(target:)
Dir.chdir(File.dirname(target)) do
File.open(File.basename(target), "wb") do |target_file|
decompress(url.open, target_file)
end
File.chmod(0755, target)
end

true
rescue OpenURI::HTTPError
false
end

def url
URI.parse(format(
"https://github.com/%{owner}/%{repository}/releases/download/%{version}/%{filename}",
owner: owner,
repository: repository,
version: version,
filename: filename
))
end

def filename
format(
"%{basename}-%{os}-%{cpu}.%{extension}",
basename: basename,
os: platform.os,
cpu: platform.cpu,
extension: platform.os == "windows" ? "exe.gz" : "gz"
)
end

private

def decompress(source, target)
zlib = Zlib::GzipReader.new(source)
target << zlib.read
ensure
zlib.close
end
end

Platform = Struct.new(:ruby_config) do
def initialize(ruby_config = RbConfig::CONFIG)
super(ruby_config)
end

def format_path(path)
case os
when "windows"
File.extname(path) != ".exe" ? path + ".exe" : path
else
path
end
end

def to_s
format("%{os}-%{cpu}", os: os, cpu: cpu)
end

def os
case ruby_config.fetch("host_os")
when /linux/
"linux"
when /darwin/
"darwin"
else
"windows"
end
end

def cpu
case ruby_config.fetch("host_cpu")
when /arm.*64/
"arm64"
when /64/
"amd64"
else
"386"
end
end
end
end
1 change: 1 addition & 0 deletions ext/shopify-extensions/version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v0.1.0
153 changes: 153 additions & 0 deletions test/ext/shopify-extensions/shopify_extensions_test.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
require "test_helper"
require_relative "../../../ext/shopify-extensions/shopify_extensions.rb"

module ShopifyExtensions
class ShopfyExtensionsTest < Minitest::Test
def test_installation_of_existing_version_for_mac_os
stub_executable_download

target = File.join(Dir.mktmpdir, "shopify-extensions")

Install.call(
platform: Platform.new({
"host_os" => "darwin20.3.0",
"host_cpu" => "x86_64",
}),
version: "v0.1.0",
target: target
)

assert File.file?(target)
assert File.executable?(target)
assert_match(/v0.1.0/, %x(#{target}))
end

def test_installation_of_existing_version_for_windows
stub_executable_download

target = File.join(Dir.mktmpdir, "shopify-extensions.exe")

Install.call(
platform: Platform.new({
"host_os" => "mingw32",
"host_cpu" => "x64",
}),
version: "v0.1.0",
target: target
)

assert File.file?(target)
assert File.executable?(target)
end

def test_handle_http_errors_during_asset_download
simulate_broken_asset_link

target = File.join(Dir.mktmpdir, "shopify-extensions")

assert_raises(InstallationError) do
Install.call(
platform: Platform.new({
"host_os" => "darwin20.3.0",
"host_cpu" => "x86_64",
}),
version: "v0.1.0",
target: target
)
end
end

def test_incorrect_binary
stub_executable_download

target = File.join(Dir.mktmpdir, "shopify-extensions.exe")
File.expects(:executable?).with(target).returns(false)

error = assert_raises(InstallationError) do
Install.call(
platform: Platform.new({
"host_os" => "mingw32",
"host_cpu" => "x64",
}),
version: "v0.1.0",
target: target
)
end

assert_equal "Failed to install shopify-extensions properly", error.message
end

class PlatformTest < MiniTest::Test
def test_recognizes_linux
linux_vm = ruby_config(os: "linux-gnu", cpu: "x86_64")
assert_equal "linux-amd64", Platform.new(linux_vm).to_s
end

def test_recognices_mac_os
t6d marked this conversation as resolved.
Show resolved Hide resolved
intel_mac = ruby_config(os: "darwin20.3.0", cpu: "x86_64")
m1_mac = ruby_config(os: "darwin20.3.0", cpu: "arm64")

assert_equal "darwin-amd64", Platform.new(intel_mac).to_s
assert_equal "darwin-arm64", Platform.new(m1_mac).to_s
end

def test_recognices_windows
t6d marked this conversation as resolved.
Show resolved Hide resolved
windows_vm_64_bit = ruby_config(os: "mingw32", cpu: "x64")
windows_vm_32_bit = ruby_config(os: "mingw32", cpu: "i686")
assert_equal "windows-amd64", Platform.new(windows_vm_64_bit).to_s
assert_equal "windows-386", Platform.new(windows_vm_32_bit).to_s
end

def test_adds_exe_extension_to_binaries_on_windows
windows_vm_64_bit = ruby_config(os: "mingw32", cpu: "x64")
assert_equal "some/command.exe", Platform.new(windows_vm_64_bit).format_path("some/command")
end

private

def ruby_config(os:, cpu:)
{
"host_os" => os,
"host_cpu" => cpu,
}
end
end

def stub_executable_download
dummy_archive = load_dummy_archive

stub_request(:get, "https://github.com/Shopify/shopify-cli-extensions/releases/download/v0.1.0/shopify-extensions-windows-amd64.exe.gz")
.to_return(
status: 200,
headers: {
"Content-Type" => "application/octet-stream",
"Content-Disposition" => "attachment; filename=shopify-extensions-windows-amd64.exe.gz",
"Content-Length" => dummy_archive.size,
},
body: dummy_archive
)

stub_request(:get, "https://github.com/Shopify/shopify-cli-extensions/releases/download/v0.1.0/shopify-extensions-darwin-amd64.gz")
.to_return(
status: 200,
headers: {
"Content-Type" => "application/octet-stream",
"Content-Disposition" => "attachment; filename=shopify-extensions-darwin-amd64.gz",
"Content-Length" => dummy_archive.size,
},
body: dummy_archive
)
end

def simulate_broken_asset_link
stub_request(:get, "https://github.com/Shopify/shopify-cli-extensions/releases/download/v0.1.0/shopify-extensions-darwin-amd64.gz")
.to_raise(OpenURI::HTTPError.new("404 Not Found", StringIO.new))
end

def load_dummy_archive
path = File.expand_path("../../../fixtures/shopify-extensions.gz", __FILE__)
raise "Dummy archive not found: #{path}" unless File.file?(path)
File.read(path)
end
end
end
Binary file added test/fixtures/shopify-extensions.gz
Binary file not shown.