WIP: Experimental no-TLS version #77

ethomson · 2018-10-07T12:37:26Z

Build a version without HTTPS support to support built-in HTTPS.

Starting this so that I can get a package built by the CI systems.

bording · 2018-10-07T16:08:58Z

Checking for module 'libcurl'
--   Found libcurl, version 7.35.0
--   Resolved libraries: /usr/lib/x86_64-linux-gnu/libcurl.so
-- Could NOT find HTTP_Parser (missing: HTTP_PARSER_INCLUDE_DIR HTTP_PARSER_LIBRARY) 
-- http-parser version 2 was not found or disabled; using bundled 3rd-party sources.
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.8") 
-- LIBSSH2 not found. Set CMAKE_PREFIX_PATH if it is installed outside of the default search path.
-- Enabled features:
 * tracing, tracing support
 * threadsafe, threadsafe support
 * cURL, cURL for HTTP proxy support
 * SHA, using CollisionDetection
 * http-parser, http-parser support (bundled)
 * zlib, using system zlib
-- Disabled features:
 * debugpool, debug pool allocator
 * HTTPS, no support
 * SSH, SSH transport support
 * SPNEGO, SPNEGO authentication support
 * iconv, iconv encoding conversion support

Does that mean it's still linking against libcurl even when HTTPS is off? What about zlib? Seems like if we want to avoid any native dependencies, we'd want to not use that was well, right?

bording · 2018-10-07T16:11:37Z

.travis.yml

@@ -41,6 +25,6 @@ before_install:

 install: true

-script: if [[ $RID == "osx" ]]; then ./build.libgit2.sh ; else ./dockerbuild.sh ; fi
+script: ./build.libgit2.sh


I assume this is a temporary change? I would think we'd want to keep building things in docker to be able to better control things?

Also, we'll likely need to keep the alpine build (with tweaks to the RID) because of needing musl libc binary as well.

ethomson · 2018-10-08T09:52:50Z

Does that mean it's still linking against libcurl even when HTTPS is off? What about zlib? Seems like if we want to avoid any native dependencies, we'd want to not use that was well, right?

That's right. I was working on a PoC and turning off TLS was the important bits, so that I made sure that I didn't have any false positives.

I assume this is a temporary change? I would think we'd want to keep building things in docker to be able to better control things?

No idea. This is just a spike, and if we decide to move it forward, even as a preview option, then I think that we are going to need to incorporate this into the main build. That means that there are a lot of build integration questions that need to get answered.

Also, we'll likely need to keep the alpine build (with tweaks to the RID) because of needing musl libc binary as well.

That would be disappointing. We shouldn't need a separate binary for alternative libc's. Do you know what problems we were having with interoperability?

bording · 2018-10-08T22:10:36Z

That would be disappointing. We shouldn't need a separate binary for alternative libc's. Do you know what problems we were having with interoperability?

I was under the impression, perhaps mistakenly, that this would be a requirement. Would a binary compiled on Ubuntu work on Alpine?

Using the binaries in the 235 release, I can see some differences in the ldd output between the alpine-x64 and linux-x64 binaries. (I did sort both of these to line up all the entries)

alpine-x64:

        linux-vdso.so.1 (0x00007fffdfa13000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0460f70000)
        libcurl.so.4 => /usr/lib/x86_64-linux-gnu/libcurl.so.4 (0x00007f0463370000)
        libssl.so.1.0.0 => not found
        libcrypto.so.1.0.0 => not found
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f0463150000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0460bd0000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f0463a00000)
        libc.musl-x86_64.so.1 => not found
        libnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f0462f10000)
        libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f0462ce0000)
        librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f0462ac0000)
        libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f0462890000)
        libpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f0462680000)
        libssl.so.1.0.2 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2 (0x00007f0462410000)
        libcrypto.so.1.0.2 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2 (0x00007f0461f90000)
        libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f0461d40000)
        libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f0461a60000)
        libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f0461820000)
        libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f0461610000)
        liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f0461400000)
        libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f0461190000)
        libunistring.so.0 => /usr/lib/x86_64-linux-gnu/libunistring.so.0 (0x00007f04608b0000)
        libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f0460510000)
        libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f04602d0000)
        libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f0460080000)
        libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f045fdf0000)
        libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f045fae0000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f045f8d0000)
        libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f045f6c0000)
        libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f045f4b0000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f045f280000)
        libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f045f060000)
        libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f045edf0000)
        libidn.so.11 => /lib/x86_64-linux-gnu/libidn.so.11 (0x00007f045ebb0000)
        libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f045e990000)
        libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f045e760000)
        libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f045e550000)

linux-x64:

        linux-vdso.so.1 (0x00007ffff540b000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f8dec750000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8dec530000)
        libcurl.so.4 => /usr/lib/x86_64-linux-gnu/libcurl.so.4 (0x00007f8dec2b0000)
        libssl.so.1.0.0 => not found
        libcrypto.so.1.0.0 => not found
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f8dec080000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8debce0000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f8dece00000)
        libnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f8debab0000)
        libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f8deb880000)
        librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f8deb660000)
        libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f8deb430000)
        libpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f8deb210000)
        libssl.so.1.0.2 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2 (0x00007f8deafa0000)
        libcrypto.so.1.0.2 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2 (0x00007f8deab30000)
        libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f8dea8e0000)
        libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f8dea600000)
        libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f8dea3c0000)
        libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f8dea1a0000)
        liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f8de9f90000)
        libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f8de9d30000)
        libunistring.so.0 => /usr/lib/x86_64-linux-gnu/libunistring.so.0 (0x00007f8de9a10000)
        libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f8de9670000)
        libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f8de9430000)
        libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f8de91e0000)
        libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f8de8f50000)
        libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f8de8c40000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8de8a30000)
        libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f8de8820000)
        libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f8de8610000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f8de83e0000)
        libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f8de81c0000)
        libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f8de7f50000)
        libidn.so.11 => /lib/x86_64-linux-gnu/libidn.so.11 (0x00007f8de7d10000)
        libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f8de7af0000)
        libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f8de78c0000)
        libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f8de76b0000)

bording · 2018-10-08T23:39:26Z

In comparison, the linux binary built from this PR looks much leaner already:

        linux-vdso.so.1 (0x00007ffffbe39000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fe456230000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe456010000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe455c70000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fe456800000)

ldd -u on it seems to indicate there are some unused dependencies:

Unused direct dependencies:
        /lib/x86_64-linux-gnu/librt.so.1
        /lib/x86_64-linux-gnu/libpthread.so.0

tmat · 2018-10-25T23:46:56Z

@bording @ethomson Do you guys have ETA for finishing this work (and libgit2/libgit2sharp#1618)?

BTW, there is also dotnet/sourcelink#197

bording · 2018-10-26T01:31:14Z

@tmat Regarding the RHEL6 problem, it's hard to test against RHEL since I don't have access to it. The current RHEL binary is actually built with CentOS 7. Apparently the glibc version on RHEL6 is too old.

I guess as part of this work we need to ensure that the version of glibc we compile against is compatible with all of the distros on https://github.com/dotnet/core/blob/master/release-notes/2.1/2.1-supported-os.md

ethomson · 2018-11-06T14:42:14Z

@bording @ethomson Do you guys have ETA for finishing this work (and libgit2/libgit2sharp#1618)?

Nope! Please remember that this is a hobby project, it's done by people in their spare time, which none of us really have enough of anyway. If you're using this at your employer, in production code then you could help contributing!

Build a version without HTTPS support to support built-in HTTPS.

Update to 0.27.7 with callback fixes cherry-picked.

ethomson · 2018-11-09T07:49:53Z

Also, @tmat, if you don't need any networking in your application then you can use this today. All that's left here is to add networking back in, which is seeming sort of painful given some of the changes between .NET Core and .NET Framework.

tmat · 2018-11-09T13:40:44Z

@ethomson I'd love to help and contribute, as I did before, but unfortunately I lack expertise in this area. I definitely understand this is not your full time job and didn't mean to push. Just wanted to know some kind of estimate to adjust our plans accordingly.

if you don't need any networking in your application then you can use this today.

This is great to hear as we do not need networking at all. So, you're saying that the latest released build of LibGit2Sharp doesn't have dependencies that are only available on some Linux distros unless networking API is used? If so then that would resolve all our issues and we could indeed use it.

bording · 2018-11-09T15:45:23Z

Also, @tmat, if you don't need any networking in your application then you can use this today.

I would like to add a caveat that I'm still not sure the single linux binary being built here is actually going to just work on all distros yet.

ethomson · 2018-11-09T17:59:32Z

@ethomson I'd love to help and contribute, as I did before, but unfortunately I lack expertise in this area. I definitely understand this is not your full time job and didn't mean to push. Just wanted to know some kind of estimate to adjust our plans accordingly.

No worries. This is my top priority for libgit2/LibGit2Sharp, but it's hard to have an ETA.

This is great to hear as we do not need networking at all. So, you're saying that the latest released build of LibGit2Sharp doesn't have dependencies that are only available on some Linux distros unless networking API is used? If so then that would resolve all our issues and we could indeed use it.

No, but I can create a prerelease of a new nuget package. I'm hesitant to release a new "LibGit2Sharp" package that drastically changes the way networking works in ways that aren't well tested. So I think that I'll create a new package entirely. I was thinking "LibGit2Sharp.Lite" since it's unencumbered by additional native binaries. (But - like 4.4BSD Lite - it sort of sounds like it's got reduced functionality, even though it doesn't. 🤔)

I would like to add a caveat that I'm still not sure the single linux binary being built here is actually going to just work on all distros yet.

My goal is to only take a dependency on libc, so I expect it would work anywhere with ELF support. 😀

bording · 2018-11-09T18:18:13Z

My goal is to only take a dependency on libc, so I expect it would work anywhere with ELF support. 😀

I'm still not sure that's true. For example, consider dotnet/sourcelink#197 and the change I made in #79 to attempt to fix it.

The library clearly had metadata that specified "GLIBC" and a minimum version. That implies to me two things:

We have to be careful about the version of GLIBC we build against compared to the versions installed on the distros we want to support
We need to see what happens when a library compiled against GLIBC runs on a system with musl libc (alpine)

That's why I still want to do some more investigation and testing.

bording · 2018-11-09T18:19:56Z

So I think that I'll create a new package entirely. I was thinking "LibGit2Sharp.Lite" since it's unencumbered by additional native binaries. (But - like 4.4BSD Lite - it sort of sounds like it's got reduced functionality, even though it doesn't. 🤔)

I'm not really sure that's a good idea either, since it would be a temporary thing, right? If the changes here are made, we would only be supporting the "Lite" version, so which of the two LibGit2Sharp packages survives?

ethomson · 2018-11-09T18:23:00Z

I'm not really sure that's a good idea either, since it would be a temporary thing, right? If the changes here are made, we would only be supporting the "Lite" version, so which of the two LibGit2Sharp packages survives?

I think that there are going to be tradeoffs with both, so I don't think it's crazy to think that we could support both in perpetuity.

bording · 2018-11-09T18:28:28Z

I think that there are going to be tradeoffs with both, so I don't think it's crazy to think that we could support both in perpetuity.

Hmm, given all the headaches around the native dependencies, I really wasn't considering keeping them around as a viable option.

ethomson · 2018-11-09T18:31:28Z

Hmm, given all the headaches around the native dependencies, I really wasn't considering keeping them around as a viable option.

There's no doubt that this is going to be slower, what's unknown is by how much. And there's no win here on Windows: libgit2 on Windows uses the same libraries that .NET does, so this just adds several layers of indirection to get there.

I think that this is the Right Thing for people who don't do any networking, or only a limited amount of networking. It's very possible that it's only the right thing in those circumstances and that everybody else will want to use the existing setup with .NET telling the native code to go do a lot of work and come back when it's done instead of marshalling every byte back and forth.

bording · 2018-11-09T18:57:54Z

@ethomson I get the motivation from the performance perspective, I'm just not sure keeping the linux dependencies around is a viable option that actually works for anyone.

The problem with continuing to provide linux binaries as they currently are is that if you aren't just using LibGit2Sharp from something like a console or web app, then it's up to the consumer to understand all of the complicated RID loading logic to get the "right" library, and I'm not sure there's much we can do to fix that. For example, anyone currently trying to use LibGit2Sharp from an MSBuild task is going to be hating life as soon as they want to support .NET Core.

It's also incredibly hard for someone to even figure out what the problem is when it doesn't work. It just seems like LibGit2Sharp (or more likely the thing they are trying to use that uses LibGit2Sharp behind the scenes) is broken and they have no idea why.

The current RID selection was made based on my trying to optimize for the most coverage with the minimum amount of binaries, but me not being able to get access to every single distro and version of a distro means I've had to make some assumptions that have not always turned out to be true.

To really cover things 100%, we'd need to have a separate binary for every single version of every single distro, and that's just not going to be acceptable to anyone.

We also run into problems where the runtime.json that controls the RIDs .NET Core knows about isn't as updated as we need it to be to be able to target things the way we might need to, so that's another source of problems.

And that doesn't even begin to consider mono support where we don't even have a way to automatically choose the correct linux binary.

I wonder if we could so some sort of hybrid thing where Windows and maybe macOS continue to still be built with the native support and only turn it off for the linux builds where the native dependencies are nothing but a pit of hurting and pain?

ethomson · 2018-11-10T11:51:02Z

I wonder if we could so some sort of hybrid thing where Windows and maybe macOS continue to still be built with the native support and only turn it off for the linux builds where the native dependencies are nothing but a pit of hurting and pain?

Indeed, that's a possibility.

alex-weaver · 2018-11-11T14:14:42Z

I am totally behind the idea of requiring the user to supply the native binaries. Right now I'm trying to build libgit2sharp with SSH support and run it on .NET Core and the process is... not fun. I figure it would be considerably easier if libgit2sharp just provided the wrappers over libgit2 then its up to the library's user to provide the binaries built with whatever settings are appropriate for their use case and target platforms.

ethomson · 2018-11-11T14:16:48Z

@alex-weaver this PR does not do that, but you can do that today. The https://github.com/libgit2/libgit2sharp.nativebinaries repo should have instructions for building and packaging your own.

alex-weaver · 2018-11-11T14:21:07Z

@ethomson thanks, I'll try that. Kinda OT for this thread, but is there any intent to merge this libgit2/libgit2sharp#1072 so that users can use ssh functionality by replacing the binaries without needing to modify libgit2sharp itself?

ethomson · 2018-11-11T14:39:55Z

@alex-weaver Not as-is. Making SSH support easier is a thing that we should pursue, but I don't want to be the one who's on the hook for dealing with security releases in SSH libraries. 😰

Discovery and dynamic loading seems reasonable; perhaps using a managed ssh library.

bording · 2020-04-05T03:13:26Z

Replaced by #96

ethomson force-pushed the ethomson/notls branch 2 times, most recently from f02eadb to bb1a94f Compare October 7, 2018 13:44

bording reviewed Oct 7, 2018

View reviewed changes

ethomson force-pushed the ethomson/notls branch 2 times, most recently from e4cffbb to d03ea98 Compare October 8, 2018 09:47

bording mentioned this pull request Oct 8, 2018

Cannot build on Ubuntu 18.04 TLS dotnet/roslyn#29289

Closed

kll mentioned this pull request Oct 15, 2018

GitVersion on VSTS fails with hosted macOS and Linux agents GitTools/GitVersion#1497

Closed

bording mentioned this pull request Nov 3, 2018

GetBuildVersion task fails for full Framework builds dotnet/Nerdbank.GitVersioning#228

Closed

ethomson force-pushed the ethomson/notls branch from 2d61e2f to 976c180 Compare November 6, 2018 14:32

Experimental no-TLS version

3071305

Build a version without HTTPS support to support built-in HTTPS.

ethomson force-pushed the ethomson/notls branch from a4e0231 to 6bde9d8 Compare November 6, 2018 22:59

Update libgit2 to 5851981

6092bcd

Update to 0.27.7 with callback fixes cherry-picked.

ethomson force-pushed the ethomson/notls branch from 6bde9d8 to 6092bcd Compare November 6, 2018 23:53

alex-weaver mentioned this pull request Nov 11, 2018

Question: SSH Support libgit2/libgit2sharp#1422

Closed

bording mentioned this pull request Nov 28, 2018

Update libgit2 to a904fc6 #82

Merged

bording mentioned this pull request Apr 20, 2019

Stop depending on OpenSSL on linux #96

Merged

bording closed this Apr 5, 2020

bording deleted the ethomson/notls branch April 5, 2020 03:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Experimental no-TLS version #77

WIP: Experimental no-TLS version #77

ethomson commented Oct 7, 2018

bording commented Oct 7, 2018

bording Oct 7, 2018

ethomson commented Oct 8, 2018

bording commented Oct 8, 2018

bording commented Oct 8, 2018

tmat commented Oct 25, 2018

bording commented Oct 26, 2018

ethomson commented Nov 6, 2018

ethomson commented Nov 9, 2018

tmat commented Nov 9, 2018 •

edited

Loading

bording commented Nov 9, 2018

ethomson commented Nov 9, 2018

bording commented Nov 9, 2018

bording commented Nov 9, 2018

ethomson commented Nov 9, 2018

bording commented Nov 9, 2018

ethomson commented Nov 9, 2018

bording commented Nov 9, 2018 •

edited

Loading

ethomson commented Nov 10, 2018

alex-weaver commented Nov 11, 2018

ethomson commented Nov 11, 2018

alex-weaver commented Nov 11, 2018

ethomson commented Nov 11, 2018

bording commented Apr 5, 2020

WIP: Experimental no-TLS version #77

WIP: Experimental no-TLS version #77

Conversation

ethomson commented Oct 7, 2018

bording commented Oct 7, 2018

bording Oct 7, 2018

Choose a reason for hiding this comment

ethomson commented Oct 8, 2018

bording commented Oct 8, 2018

bording commented Oct 8, 2018

tmat commented Oct 25, 2018

bording commented Oct 26, 2018

ethomson commented Nov 6, 2018

ethomson commented Nov 9, 2018

tmat commented Nov 9, 2018 • edited Loading

bording commented Nov 9, 2018

ethomson commented Nov 9, 2018

bording commented Nov 9, 2018

bording commented Nov 9, 2018

ethomson commented Nov 9, 2018

bording commented Nov 9, 2018

ethomson commented Nov 9, 2018

bording commented Nov 9, 2018 • edited Loading

ethomson commented Nov 10, 2018

alex-weaver commented Nov 11, 2018

ethomson commented Nov 11, 2018

alex-weaver commented Nov 11, 2018

ethomson commented Nov 11, 2018

bording commented Apr 5, 2020

tmat commented Nov 9, 2018 •

edited

Loading

bording commented Nov 9, 2018 •

edited

Loading