environment vars on Windows should be case-insensitive

[tkf]

Disclaimer: I don't personally use Windows and I inferred this behavior only through appveyor. So maybe I'm missing some points here.

Anyway, how do you update PATH to be used for setenv(command::Cmd, env) on Windows? For example, if I do

env_var = copy(ENV)
env_var["PATH"] = "some_new_path" * ';' * env_var["PATH"]
setenv(command::Cmd, env_var)

it doesn't work since the original PATH is in env_var["Path"] not env_var["PATH"]. I had to implement something like this (JuliaPy/Conda.jl#121 (comment)):

env_var = copy(ENV)
all_keys = collect(keys(env_var))
ikey = findfirst(x -> uppercase(x) == "PATH", all_keys)
has_key = ikey !== nothing
path_key = has_key ? all_keys[ikey] : "PATH"
if has_key
    env_var[path_key] = extra_path * path_sep * env_var[path_key]
else
    env_var[path_key] = extra_path
end

I tried empty!(ENV); merge!(ENV, env_var) but apparently empty!(ENV) is not implemented (at least on Linux). Also, it would have emitted bunch of syscalls so maybe not a good idea in the first place.

So, I thought maybe Base needs to have a case-insensitive (in-memory) dictionary and copy(ENV) on Windows should return it. It also facilitates cross-platform testing.

cc: @stevengj

[stevengj]

It seems like Base.EnvDict should treat the keys case-insensitively on Windows:

• Make key lookups (getindex, haskey, etcetera) case-insensitive.

• Always display the keys as uppercase

• copy(ENV) should return a Dict with uppercase keys.

Or does it do this already? I don't have a Windows machine handy to check.

[stevengj]

See e.g. dotnet/corefx#13146 and cucumber/aruba#291 for related discussions in other languages.

[tkf]

I thought ENV was already case-insensitive as it just wraps syscalls. For example getindex just calls:

julia/base/env.jl

Lines 9 to 23 in fa7d1a9

	function access_env(onError::Function, str::AbstractString)
	var = cwstring(str)
	len = _getenvlen(var)
	if len == 0
	return Libc.GetLastError() != ERROR_ENVVAR_NOT_FOUND ? "" : onError(str)
	end
	val = zeros(UInt16,len)
	ret = ccall(:GetEnvironmentVariableW,stdcall,UInt32,(Ptr{UInt16},Ptr{UInt16},UInt32),var,val,len)
	if (ret == 0 && len != 1) || ret != len-1 || val[end] != 0
	error(string("getenv: ", str, ' ', len, "-1 != ", ret, ": ", Libc.FormatMessage()))
	end
	pop!(val) # NUL
	return transcode(String, val)
	end

The problem I wanted to brought up here is:

• What (should) happen for run(setenv(..., Dict("PATH"=>"...", "Path"=>"...")))?
• Similarly, what should happen when merge!(ENV, env) when env has the overlapping keys?

copy(ENV) should return a Dict with uppercase keys.

Yes, I think this is a good idea (on Windows). (On Linux, I see that copy(ENV) calls copy(::AbstractDict) (which is fine on Linux). Not sure if it dispatches to other method in Windows.)

[tkf]

merge!(ENV, Dict("key" => "1", "KEY" => "2")) doesn't throw:
https://ci.appveyor.com/project/tkf/myplayground-jl/build/job/4pej3sggfba38gc8#L98

setenv(cmd, env) doesn't throw when env has overlapping keys:
https://ci.appveyor.com/project/tkf/myplayground-jl/build/job/4pej3sggfba38gc8#L110

Non-uppercase environment variables:

setdiff(keys(ENV), Set(map(uppercase, collect(keys(ENV))))) = Set(["EnableNuGetPackageRestore", "ProgramFiles(x86)", "ProgramData", "SystemRoot", "CommonProgramW6432", "CodeContractsInstallDir", "windir", "ProgramW6432", "ComSpec", "SystemDrive", "ProgramFiles", "VSSDK140Install", "SomeKey", "VSSDK120Install", "julia_version", "CommonProgramFiles", "ChocolateyInstall", "Path", "CommonProgramFiles(x86)", "lastexitcode", "Platform", "xunit20", "ChocolateyLastPathUpdate", "7zip", "PSModulePath"])
https://ci.appveyor.com/project/tkf/myplayground-jl/build/job/4pej3sggfba38gc8#L108

code: https://github.com/tkf/MyPlayground.jl/blob/32ee7fce51a7512d0e2a91a4123278e4a9f3965a/test/runtests.jl

[stevengj]

What case does keys(ENV) return? My impression was that the GetEnvironmentVariables is case-preserving even though getindex is case-insensitive. If so, I think these should be uppercased on windows.

[tkf]

What case does keys(ENV) return?

Not the direct answer, but see the output of setdiff(keys(ENV), Set(map(uppercase, collect(keys(ENV)))) above.

My impression was that the GetEnvironmentVariables is case-preserving even though getindex is case-insensitive.

Yeah, it looks like so. Here is the output of display(ENV):

Base.EnvDict with 102 entries:
  "7zip" => "\"C:\\Program Files\\7-Zip\\7z.exe\""
  "=::" => "::\\"
  "=C:" => "C:\\projects\\myplayground-jl\\test"
  "ALLUSERSPROFILE" => "C:\\ProgramData"
  "APPDATA" => "C:\\Users\\appveyor\\AppData\\Roaming"
  "APPVEYOR" => "True"
  "APPVEYOR_ACCOUNT_NAME" => "tkf"
  "APPVEYOR_API_URL" => "http://localhost:1035/"
  "APPVEYOR_BUILD_AGENT_HYPERV_NIC_CONFIGURED" => "true"
  "APPVEYOR_BUILD_FOLDER" => "C:\\projects\\myplayground-jl"
  "APPVEYOR_BUILD_ID" => "19015984"
  "APPVEYOR_BUILD_NUMBER" => "2"
  "APPVEYOR_BUILD_VERSION" => "1.0.2"
  "APPVEYOR_BUILD_WORKER_IMAGE" => "Visual Studio 2015"
  "APPVEYOR_JOB_ID" => "4pej3sggfba38gc8"
  "APPVEYOR_JOB_NAME" => "Environment: julia_version=1; Platform: x86"
  "APPVEYOR_JOB_NUMBER" => "1"
  "APPVEYOR_PROJECT_ID" => "495883"
  "APPVEYOR_PROJECT_NAME" => "MyPlayground.jl"
  "APPVEYOR_PROJECT_SLUG" => "myplayground-jl"
  "APPVEYOR_REPO_BRANCH" => "appveyor/main"
  "APPVEYOR_REPO_COMMIT" => "32ee7fce51a7512d0e2a91a4123278e4a9f3965a"
  "APPVEYOR_REPO_COMMIT_AUTHOR" => "Takafumi Arakaki"
  "APPVEYOR_REPO_COMMIT_AUTHOR_EMAIL" => "[email protected]"
  "APPVEYOR_REPO_COMMIT_MESSAGE" => "Test ENV canse-(in)sensitivity"
  "APPVEYOR_REPO_COMMIT_TIMESTAMP" => "2018-09-24T22:14:51.0000000Z"
  "APPVEYOR_REPO_NAME" => "tkf/MyPlayground.jl"
  "APPVEYOR_REPO_PROVIDER" => "gitHub"
  "APPVEYOR_REPO_SCM" => "git"
  "APPVEYOR_REPO_TAG" => "false"
  "APPVEYOR_URL" => "https://ci.appveyor.com"
  "APR_ICONV_PATH" => "C:\\Program Files (x86)\\Subversion\\iconv"
  "AVVM_DOWNLOAD_URL" => "https://appveyordownloads.blob.core.windows.net/avvm"
  "ChocolateyInstall" => "C:\\ProgramData\\chocolatey"
  "ChocolateyLastPathUpdate" => "Fri Jan 19 03:37:46 2018"
  "CI" => "True"
  "CI_LINUX" => "False"
  "CI_WINDOWS" => "True"
  "CodeContractsInstallDir" => "C:\\Program Files (x86)\\Microsoft\\Contracts\\"
  "CommonProgramFiles

https://ci.appveyor.com/project/tkf/myplayground-jl/build/job/4pej3sggfba38gc8#L57