Symlinks not preserved #112

chrispcampbell · 2018-01-05T21:48:26Z

Issue

This issue sounds similar to #106, but that ticket seemed to cover a different scenario, so I figured I'd file this as a separate issue.

I just updated our macOS build to use Rome and found that the build would break in code signing, but only in the case where binaries are copied in from Rome's local cache. (The build was successful when using artifacts produced directly by Carthage.) I discovered that the frameworks produced by Carthage contain symlinks, for example:

% ls -la Carthage/Build/Mac/CocoaLumberjack.framework
total 32
drwxr-xr-x   7 campbell  staff  238 Jan  5 11:31 .
drwxr-xr-x  19 campbell  staff  646 Jan  5 13:27 ..
lrwxr-xr-x   1 campbell  staff   32 Jan  5 11:31 CocoaLumberjack -> Versions/Current/CocoaLumberjack
lrwxr-xr-x   1 campbell  staff   24 Jan  5 11:31 Headers -> Versions/Current/Headers
lrwxr-xr-x   1 campbell  staff   24 Jan  5 11:31 Modules -> Versions/Current/Modules
lrwxr-xr-x   1 campbell  staff   26 Jan  5 11:31 Resources -> Versions/Current/Resources
drwxr-xr-x   4 campbell  staff  136 Jan  5 11:31 Versions

After a rome upload step, if I unzip the relevant file from the Rome cache, I see that the symlinks are no longer present:

% cd ~/Library/Caches/Rome/CocoaLumberjack/Mac/
% unzip CocoaLumberjack.framework-3.3.0.zip
% ls -la Carthage/Build/Mac/CocoaLumberjack.framework
total 504
drwxr-xr-x   7 campbell  staff     238 Jan  5  2018 .
drwxr-xr-x   3 campbell  staff     102 Jan  5 13:29 ..
-rwxr-xr-x   1 campbell  staff  255500 Jan  5  2018 CocoaLumberjack
drwxr-xr-x  18 campbell  staff     612 Jan  5  2018 Headers
drwxr-xr-x   3 campbell  staff     102 Jan  5  2018 Modules
drwxr-xr-x   3 campbell  staff     102 Jan  5  2018 Resources
drwxr-xr-x   4 campbell  staff     136 Jan  5  2018 Versions

So if later I use rome download, the Carthage/Build directory will be populated with the framework(s) that don't contain symlinks.

This causes codesign to fail later with a "bundle format is ambiguous" error, which is described here:
https://developer.apple.com/library/content/technotes/tn2206/_index.html#//apple_ref/doc/uid/DTS40007919-CH1-TNTAG313

"Perhaps a framework was copied incorrectly so the symlinks it contained were converted to normal files."

The relevant portion of our Cartfile.resolved:

github "CocoaLumberjack/CocoaLumberjack" "3.3.0"

And our Romefile:

[Cache]
local = ~/Library/Caches/Rome

[RepositoryMap]
CocoaLumberjack = CocoaLumberjack, CocoaLumberjackSwift

Rome version: 0.13.1.35
OS and version: macOS 10.12.6

The text was updated successfully, but these errors were encountered:

tmspzz · 2018-01-08T09:26:41Z

Sounds like a bug. I'll try to fix it asap.

tmspzz · 2018-01-11T14:24:06Z

Ok, small update:

I haven't forgot this.
this is an issue with the zip library
I'm trying to patch the zip library (you would not believe how messed up zip is)
if I don't succeed I will just used zip form the command line

tmspzz · 2018-01-12T13:16:24Z

@chrispcampbell Please try pre-release https://github.com/blender/Rome/releases/tag/v0.14.0.36

I haven't tried to build but the contents of the archive seem to preserve symlinks

tmspzz · 2018-01-16T11:55:33Z

@chrispcampbell any feedback on this?

chrispcampbell · 2018-01-16T18:56:55Z

Thanks for the fix, I will try it out today and will let you know!

chrispcampbell · 2018-01-17T06:16:13Z

@blender I tested with v0.14.0.36. The good news is that it appears to partially preserve symlinks when doing a rome upload to the local Rome cache. Like, after a rome upload I can unzip a zip file (in Finder) in the Rome cache and see that the unzipped contents contain symlinks. So, that's a start.

However, when I clear out my Carthage/Build directory and then do a rome download for that framework, it gives an error. First, this is what it looks like as built by Carthage:

% % ls -lR Carthage/Build/Mac/CocoaLumberjack.framework
total 32
lrwxr-xr-x  1 campbell  staff   32 Jan 16 17:03 CocoaLumberjack -> Versions/Current/CocoaLumberjack
lrwxr-xr-x  1 campbell  staff   24 Jan 16 17:03 Headers -> Versions/Current/Headers
lrwxr-xr-x  1 campbell  staff   24 Jan 16 17:03 Modules -> Versions/Current/Modules
lrwxr-xr-x  1 campbell  staff   26 Jan 16 17:03 Resources -> Versions/Current/Resources
drwxr-xr-x  4 campbell  staff  136 Jan 16 17:03 Versions

Carthage/Build/Mac/CocoaLumberjack.framework/Versions:
total 8
drwxr-xr-x  6 campbell  staff  204 Jan 16 17:03 A
lrwxr-xr-x  1 campbell  staff    1 Jan 16 17:03 Current -> A

Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A:
total 504
-rwxr-xr-x   1 campbell  staff  255500 Jan 16 17:03 CocoaLumberjack
drwxr-xr-x  18 campbell  staff     612 Jan 16 17:03 Headers
drwxr-xr-x   3 campbell  staff     102 Jan 16 17:03 Modules
drwxr-xr-x   3 campbell  staff     102 Jan 16 17:03 Resources

Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/Headers:
total 248
<stuff>

Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/Modules:
total 8
-rw-r--r--  1 campbell  staff  679 Jan 16 17:03 module.modulemap

Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/Resources:
total 8
-rw-r--r--  1 campbell  staff  1249 Jan 16 17:03 Info.plist

Then, testing the upload to an empty Rome cache:

% ls -la ~/Library/Caches/Rome
ls: /Users/campbell/Library/Caches/Rome: No such file or directory

% ~/Downloads/rome --version
0.14.0.36 - Romam uno die non fuisse conditam.

% ~/Downloads/rome upload CocoaLumberjack
Copied CocoaLumberjack.framework to: /Users/campbell/Library/Caches/Rome/CocoaLumberjack/Mac/CocoaLumberjack.framework-3.3.0.zip
Copied CocoaLumberjack.dSYM to: /Users/campbell/Library/Caches/Rome/CocoaLumberjack/Mac/CocoaLumberjack.framework.dSYM-3.3.0.zip
Copied CocoaLumberjackSwift.framework to: /Users/campbell/Library/Caches/Rome/CocoaLumberjack/Mac/CocoaLumberjackSwift.framework-3.3.0.zip
Copied CocoaLumberjackSwift.dSYM to: /Users/campbell/Library/Caches/Rome/CocoaLumberjack/Mac/CocoaLumberjackSwift.framework.dSYM-3.3.0.zip
Copied .CocoaLumberjack.version to: /Users/campbell/Library/Caches/Rome/CocoaLumberjack/.CocoaLumberjack.version-3.3.0

Next, I moved the Carthage/Build directory aside and attempted a rome download, which gives the error:

% mv Carthage/Build Carthage/Build-Old

% ~/Downloads/rome download CocoaLumberjack
Found CocoaLumberjack in local cache at: /Users/campbell/Library/Caches/Rome/CocoaLumberjack/Mac/CocoaLumberjack.framework-3.3.0.zip
rome: CRC32Mismatch "Carthage/Build/Mac/CocoaLumberjack.framework/CocoaLumberjack"

It's worth noting that I see similar CRC-related messages when I try to unzip the thing from the command line:

% unzip ~/Library/Caches/Rome/CocoaLumberjack/Mac/CocoaLumberjack.framework-3.3.0.zip -d /tmp/rometest
Archive:  /Users/campbell/Library/Caches/Rome/CocoaLumberjack/Mac/CocoaLumberjack.framework-3.3.0.zip
   creating: /tmp/rometest/Carthage/Build/Mac/CocoaLumberjack.framework/
    linking: /tmp/rometest/Carthage/Build/Mac/CocoaLumberjack.framework/CocoaLumberjack  -> Versions/Current/CocoaLumberjack  bad CRC 0933b4ab  (should be 29929617)
    linking: /tmp/rometest/Carthage/Build/Mac/CocoaLumberjack.framework/Headers  -> Versions/Current/Headers  bad CRC 8ead94f3  (should be 00000000)
    linking: /tmp/rometest/Carthage/Build/Mac/CocoaLumberjack.framework/Modules  -> Versions/Current/Modules  bad CRC 571998a1  (should be 00000000)
    linking: /tmp/rometest/Carthage/Build/Mac/CocoaLumberjack.framework/Resources  -> Versions/Current/Resources  bad CRC 8702472c  (should be 00000000)
   creating: /tmp/rometest/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/
   creating: /tmp/rometest/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/
  inflating: /tmp/rometest/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/CocoaLumberjack  
   creating: /tmp/rometest/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/Headers/
  inflating: /tmp/rometest/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/Headers/CLIColor.h  
  inflating: /tmp/rometest/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/Headers/CocoaLumberjack.h  
<more headers...>
   creating: /tmp/rometest/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/Modules/
  inflating: /tmp/rometest/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/Modules/module.modulemap  
   creating: /tmp/rometest/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/Resources/
  inflating: /tmp/rometest/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/Resources/Info.plist  
    linking: /tmp/rometest/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/Current  -> A  bad CRC d3d99e8b  (should be 00000000)
finishing deferred symbolic links:
  /tmp/rometest/Carthage/Build/Mac/CocoaLumberjack.framework/CocoaLumberjack -> Versions/Current/CocoaLumberjack
  /tmp/rometest/Carthage/Build/Mac/CocoaLumberjack.framework/Headers -> Versions/Current/Headers
  /tmp/rometest/Carthage/Build/Mac/CocoaLumberjack.framework/Modules -> Versions/Current/Modules
  /tmp/rometest/Carthage/Build/Mac/CocoaLumberjack.framework/Resources -> Versions/Current/Resources
  /tmp/rometest/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/Current -> A

tmspzz · 2018-01-17T07:12:12Z

I think I know why. Thanks for testing. Back on it.

tmspzz · 2018-01-17T11:42:34Z

@chrispcampbell

please try https://github.com/blender/Rome/releases/tag/v0.14.0.37 I have fixed CRC32 errros and I see no issue with the upload/download flow. Command line unzipping is clean too.

chrispcampbell · 2018-01-17T17:17:20Z

@blender ~~I just tried v0.14.0.37 and can confirm that the CRC32 errors are fixed. Thanks again for the fixes!~~ Actually I might have been using the wrong binary; still seeing some issues. Will add another comment once I can confirm.

chrispcampbell · 2018-01-17T17:32:12Z

@blender I just tried again and can confirm that rome upload does seem to preserve the symlinks (unzipping the zip file from the Rome cache shows correct symlinks with no CRC errors), but now the rome download step is back to not creating the symlinks when the thing gets unzipped into Carthage/Build. Any ideas?

chrispcampbell · 2018-01-17T17:54:22Z

@blender Here's the output of the unzip (for a cached zip file that was created with v0.14.0.37):

% unzip ~/Library/Caches/Rome/CocoaLumberjack/Mac/CocoaLumberjack.framework-3.3.0.zip -d /tmp/rometest4
Archive:  /Users/campbell/Library/Caches/Rome/Pup/CocoaLumberjack/Mac/CocoaLumberjack.framework-3.3.0.zip
   creating: /tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/
    linking: /tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/CocoaLumberjack  -> Versions/Current/CocoaLumberjack 
    linking: /tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Headers  -> Versions/Current/Headers 
    linking: /tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Modules  -> Versions/Current/Modules 
    linking: /tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Resources  -> Versions/Current/Resources 
   creating: /tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/
   creating: /tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/
  inflating: /tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/CocoaLumberjack  
   creating: /tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/Headers/
  inflating: /tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/Headers/CLIColor.h  
<more headers...>
   creating: /tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/Modules/
  inflating: /tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/Modules/module.modulemap  
   creating: /tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/Resources/
  inflating: /tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/Resources/Info.plist  
    linking: /tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/Current  -> A 
finishing deferred symbolic links:
  /tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/CocoaLumberjack -> Versions/Current/CocoaLumberjack
  /tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Headers -> Versions/Current/Headers
  /tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Modules -> Versions/Current/Modules
  /tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Resources -> Versions/Current/Resources
  /tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/Current -> A

And a listing of the unzipped content:

% ls -lR /tmp/rometest4
total 0
drwxr-xr-x  3 campbell  wheel  102 Jan 17 09:49 Carthage

/tmp/rometest4/Carthage:
total 0
drwxr-xr-x  3 campbell  wheel  102 Jan 17 09:49 Build

/tmp/rometest4/Carthage/Build:
total 0
drwxr-xr-x  3 campbell  wheel  102 Jan 17 09:49 Mac

/tmp/rometest4/Carthage/Build/Mac:
total 0
drwxr-xr-x  7 campbell  wheel  238 Jan 17  2018 CocoaLumberjack.framework

/tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework:
total 32

ls: CocoaLumberjack.framework/CocoaLumberjack: Permission denied
l---------  1 campbell  wheel   32 Jan 17 09:49 CocoaLumberjack

ls: CocoaLumberjack.framework/Headers: Permission denied
l---------  1 campbell  wheel   24 Jan 17 09:49 Headers

ls: CocoaLumberjack.framework/Modules: Permission denied
l---------  1 campbell  wheel   24 Jan 17 09:49 Modules

ls: CocoaLumberjack.framework/Resources: Permission denied
l---------  1 campbell  wheel   26 Jan 17 09:49 Resources
drwxr-xr-x  4 campbell  wheel  136 Jan 17  2018 Versions

/tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Versions:
total 8
drwxr-xr-x  6 campbell  wheel  204 Jan 17  2018 A

ls: Versions/Current: Permission denied
l---------  1 campbell  wheel    1 Jan 17 09:49 Current

/tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A:
total 504
-rwxr-xr-x   1 campbell  wheel  255500 Jan 17  2018 CocoaLumberjack
drwxr-xr-x  18 campbell  wheel     612 Jan 17  2018 Headers
drwxr-xr-x   3 campbell  wheel     102 Jan 17  2018 Modules
drwxr-xr-x   3 campbell  wheel     102 Jan 17  2018 Resources

/tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/Headers:
total 248
-rw-r--r--  1 campbell  wheel   1609 Jan 17  2018 CLIColor.h
<more headers...>

/tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/Modules:
total 8
-rw-r--r--  1 campbell  wheel  679 Jan 17  2018 module.modulemap

/tmp/rometest4/Carthage/Build/Mac/CocoaLumberjack.framework/Versions/A/Resources:
total 8
-rw-r--r--  1 campbell  wheel  1249 Jan 17  2018 Info.plist

Looks like it could be a permissions issue?

tmspzz · 2018-01-17T21:01:40Z

Strange. I'll investigate.

chrispcampbell · 2018-01-22T18:45:59Z

@blender Have you had a chance to look into the permissions issue?

tmspzz · 2018-01-22T21:00:41Z

Yes, I have addressed it in the zip library jgm/zip-archive#39. Will make a build asap.

tmspzz · 2018-01-22T22:51:52Z

@chrispcampbell here you go https://github.com/blender/Rome/releases/tag/v0.14.0.38

I have tested:

upload, extraction via unzip and have seen no issue from CLI
download, permission and symlinks are preserved

Let me know if it works for you as well and I will make a full release

chrispcampbell · 2018-01-23T00:29:49Z

@blender I tried v0.14.0.38 and can confirm that everything appears to be working correctly now.
I tried upload, download, and manual unzip extraction and symlinks were created correctly without permission issues. Thank you!

tmspzz · 2018-01-23T00:55:56Z

Thank you for the patience! I'll make remove the draft tag from the release and update brew too.

tmspzz added the bug label Jan 8, 2018

This was referenced Jan 11, 2018

Add ZipOption to preserve symbolic links jgm/zip-archive#37

Merged

Fix symlinks not preserved #113

Merged

c0diq mentioned this issue Jan 19, 2018

Frameworks with symlinks inside are not uploaded correctly [zlib] [common-crypto] #106

Closed

tmspzz closed this as completed Jan 23, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Symlinks not preserved #112

Symlinks not preserved #112

chrispcampbell commented Jan 5, 2018

tmspzz commented Jan 8, 2018

tmspzz commented Jan 11, 2018

tmspzz commented Jan 12, 2018

tmspzz commented Jan 16, 2018

chrispcampbell commented Jan 16, 2018

chrispcampbell commented Jan 17, 2018 •

edited

Loading

tmspzz commented Jan 17, 2018

tmspzz commented Jan 17, 2018 •

edited

Loading

chrispcampbell commented Jan 17, 2018 •

edited

Loading

chrispcampbell commented Jan 17, 2018

chrispcampbell commented Jan 17, 2018

tmspzz commented Jan 17, 2018

chrispcampbell commented Jan 22, 2018

tmspzz commented Jan 22, 2018 via email •

edited

Loading

tmspzz commented Jan 22, 2018

chrispcampbell commented Jan 23, 2018

tmspzz commented Jan 23, 2018

Symlinks not preserved #112

Symlinks not preserved #112

Comments

chrispcampbell commented Jan 5, 2018

Issue

tmspzz commented Jan 8, 2018

tmspzz commented Jan 11, 2018

tmspzz commented Jan 12, 2018

tmspzz commented Jan 16, 2018

chrispcampbell commented Jan 16, 2018

chrispcampbell commented Jan 17, 2018 • edited Loading

tmspzz commented Jan 17, 2018

tmspzz commented Jan 17, 2018 • edited Loading

chrispcampbell commented Jan 17, 2018 • edited Loading

chrispcampbell commented Jan 17, 2018

chrispcampbell commented Jan 17, 2018

tmspzz commented Jan 17, 2018

chrispcampbell commented Jan 22, 2018

tmspzz commented Jan 22, 2018 via email • edited Loading

tmspzz commented Jan 22, 2018

chrispcampbell commented Jan 23, 2018

tmspzz commented Jan 23, 2018

chrispcampbell commented Jan 17, 2018 •

edited

Loading

tmspzz commented Jan 17, 2018 •

edited

Loading

chrispcampbell commented Jan 17, 2018 •

edited

Loading

tmspzz commented Jan 22, 2018 via email •

edited

Loading