Add gitlab and gitlab-shell.

[teh]

I had to make several adjustments to make it work with nixos:

• Replace relative config file lookups with ENV variable.
• Modify gitlab-shell to not clear then environment when running
  pre-receive.
• Modify gitlab-shell to write some environment variables into
  the .authorized_keys file to make sure gitlab-shell reads the
  correct config file.

The current version requires the user to install the gitlab-shell
package into the systemPackages environment because
/run/current-system/sw/bin/gitlab-shell is hard-coded in the
.authorized_keys file.

[teh]

This is much messier than I'd like it to be. Looking forward to learning how to improve from an experienced nix person :)

[domenkozar]

Closes #2745

[offlinehacker]

Why is gemfile.lock needed?

[offlinehacker]

You could enable database by default using services.postgresql.enable = mkDefault true;. Users can still disable by setting it to false.

[teh]

done.

[teh]

For Gemfile.lock - I don't know I copied that from redmine. Checking in the gemfile seems to be the normal thing to do in the ruby world (e.g. http://stackoverflow.com/questions/4151495/should-gemfile-lock-be-included-in-gitignore)

[teh]

(rebased against master)

[teh]

I'm logging unicorn output to the syslog now in a really ugly way after wasting hours on trying to add a syslogger module: I just dumped a syslogger class directly in the unicorn config.

[offlinehacker]

Looks good, are you ok if we merge this, or are you planning to add/change something else?

[teh]

Patches: Very unlikely upstream would be interested. Running a binary from the git checkout (where all the relative imports work) seems to be the preferred mode of operation in ruby land.

I'd like to spend 2-3 more days cleaning it up a bit if that's OK?

[offlinehacker]

yup, no problem, let me know and i review again and merge this.

[gavinrogers]

@teh thanks for this. any updates? how did the cleaning up go? can i help? I've gone through similar pain myself (writing puppet module to install the omnibus versions of gitlab) so I know it might not ever be super clean :(

[teh]

@gavinrogers - I'll do a rebase tonight after work and we can merge after.

[teh]

Rebased but haven't tested latest rebase yet.

[offlinehacker]

Here's patched and tested version https://github.com/offlinehacker/nixpkgs/tree/teh-gitlab, i've also added basic nixos test.

[offlinehacker]

Where's this used and what was the idea behind?

[teh]

gitlab has a few management commands that have to be run from the command line, e.g.

sudo -u gitlab -H bundle exec rake gitlab:app:status RAILS_ENV=production

gitlab-runner is a way to run bundle code with the correct environment.

[offlinehacker]

@teh Besides these minor bugs/questions, great job, you've finished the work that many have failed before, it's not perfect, but currently good enough :) I would really like to get this into next nixos release, so with some help i can also fix and merge it myself. Currently patched version with simple nixos test is here https://github.com/offlinehacker/nixpkgs/tree/teh-gitlab

[teh]

I've tested this on a pristine server and it worked. I'm giving it one more spin right now.

Generally I'm not sure this is ready for inclusion into a "stable" system. If we ship with the next release I'd prefer there to be a big "experimental" warning somewhere!

[teh]

Rebased and fixed one more issue (permissions on clean server were wrong).

[teh]

@offlinehacker I'm away for a few days, let me know if I can do anything else!

[offlinehacker]

It's merged in 59995e1 and 13e5878. I squashed commits and summary of descriptions, i needed to fix a few things and also added a simple test(that's why second commit). I'm closing this as it's merged in.

@teh thanks again!

[lucabrunox]

Fails to build on i686: http://hydra.nixos.org/build/17868922

[domenkozar]

@teh we also need to add 32bit version of v8, should that be just manually added or could we modify the automation?

[teh]

Hm that's annoying. First I hadn't noticed that we pulled in a binary dependency specific to linux, but there also isn't a 32bit build: https://rubygems.org/gems/libv8/versions

Maybe we can disable gitlab if system is i686?

[domenkozar]

Maybe we could use the version we have packaged ourselves?

On Sat, Dec 13, 2014 at 3:59 PM, teh [email protected] wrote:

Hm that's annoying. First I hadn't noticed that we pulled in a binary
dependency specific to linux, but there also isn't a 32bit build:
https://rubygems.org/gems/libv8/versions

Maybe we can disable gitlab if system is i686?

—
Reply to this email directly or view it on GitHub
#4796 (comment).

[teh]

Looks like it's therubyracer which needs libv8.

https://github.com/cowboyd/libv8/tree/3.11#bring-your-own-v8 says we could use this:

gem install libv8 -- --with-system-v8

I'm out for today bad timing :( but will check when back tomorrow.

[offlinehacker]

I will test this now, thanks!

[offlinehacker]

Here, fix: #5323

[nbp]

Really, couldn't we load this value from a static file stored in /etc ?

[teh]

It's been a few weeks so my memory is already hazy but I think I couldn't get bundler to read the whole rails config directory from anywhere else other than the execution point (i.e. in a read-only nix-store path).

I'd be super happy if someone knowledgeable would make this cleaner!

I packaged sentry for our website and it's just so much nicer because it takes single config file on execution. None of the relative importing that rails seems to do: https://github.com/WeAreWizards/website/tree/master/sentry

[offlinehacker]

@teh do you have any idea why gitlab startup is so super slow?
On Dec 16, 2014 11:57 AM, "teh" [email protected] wrote:

In nixos/modules/services/misc/gitlab.nix
#4796 (diff):

•  wantedBy = [ "multi-user.target" ];
  

•  environment.HOME = "${cfg.stateDir}/home";
  

•  environment.UNICORN_PATH = "${cfg.stateDir}/";
  

•  environment.GITLAB_PATH = "${pkgs.gitlab}/share/gitlab/";
  

•  environment.GITLAB_APPLICATION_LOG_PATH = "${cfg.stateDir}/log/application.log";
  

•  environment.GITLAB_SATELLITES_PATH = "${cfg.stateDir}/satellites";
  

•  environment.GITLAB_SHELL_PATH = "${pkgs.gitlab-shell}";
  

•  environment.GITLAB_REPOSITORIES_PATH = "${cfg.stateDir}/repositories";
  

•  environment.GITLAB_SHELL_HOOKS_PATH = "${cfg.stateDir}/shell/hooks";
  

•  environment.BUNDLE_GEMFILE = "${pkgs.gitlab}/share/gitlab/Gemfile";
  

•  environment.GITLAB_EMAIL_FROM = "${cfg.emailFrom}";
  

•  environment.GITLAB_SHELL_CONFIG_PATH = "${cfg.stateDir}/shell/config.yml";
  

•  environment.GITLAB_SHELL_SECRET_PATH = "${cfg.stateDir}/config/gitlab_shell_secret";
  

•  environment.GITLAB_HOST = "${cfg.host}";
  

•  environment.GITLAB_DATABASE_HOST = "${cfg.databaseHost}";
  

•  environment.GITLAB_DATABASE_PASSWORD = "${cfg.databasePassword}";
  

It's been a few weeks so my memory is already hazy but I think I couldn't
get bundler to read the whole rails config directory from anywhere else
other than the execution point (i.e. in a read-only nix-store path).

I'd be super happy if someone knowledgeable would make this cleaner!

I packaged sentry for our website and it's just so much nicer because it
takes single config file on execution. None of the relative importing that
rails seems to do:
https://github.com/WeAreWizards/website/tree/master/sentry

—
Reply to this email directly or view it on GitHub
https://github.com/NixOS/nixpkgs/pull/4796/files#r21890128.

[teh]

@offlinehacker can you quantify "super slow"? E.g. we had a digital-ocean instance with gitlab and it took ~30 seconds to start on there. I think that counts as "normal" for gitlab.

[offlinehacker]

@teh 5-10 minutes in tests
On Dec 16, 2014 12:38 PM, "teh" [email protected] wrote:

In nixos/modules/services/misc/gitlab.nix
#4796 (diff):

•  wantedBy = [ "multi-user.target" ];
  

•  environment.HOME = "${cfg.stateDir}/home";
  

•  environment.UNICORN_PATH = "${cfg.stateDir}/";
  

•  environment.GITLAB_PATH = "${pkgs.gitlab}/share/gitlab/";
  

•  environment.GITLAB_APPLICATION_LOG_PATH = "${cfg.stateDir}/log/application.log";
  

•  environment.GITLAB_SATELLITES_PATH = "${cfg.stateDir}/satellites";
  

•  environment.GITLAB_SHELL_PATH = "${pkgs.gitlab-shell}";
  

•  environment.GITLAB_REPOSITORIES_PATH = "${cfg.stateDir}/repositories";
  

•  environment.GITLAB_SHELL_HOOKS_PATH = "${cfg.stateDir}/shell/hooks";
  

•  environment.BUNDLE_GEMFILE = "${pkgs.gitlab}/share/gitlab/Gemfile";
  

•  environment.GITLAB_EMAIL_FROM = "${cfg.emailFrom}";
  

•  environment.GITLAB_SHELL_CONFIG_PATH = "${cfg.stateDir}/shell/config.yml";
  

•  environment.GITLAB_SHELL_SECRET_PATH = "${cfg.stateDir}/config/gitlab_shell_secret";
  

•  environment.GITLAB_HOST = "${cfg.host}";
  

•  environment.GITLAB_DATABASE_HOST = "${cfg.databaseHost}";
  

•  environment.GITLAB_DATABASE_PASSWORD = "${cfg.databasePassword}";
  

@offlinehacker https://github.com/offlinehacker can you quantify "super
slow"? E.g. we had a digital-ocean instance with gitlab and it took ~30
seconds to start on there. I think that counts as "normal" for gitlab.

—
Reply to this email directly or view it on GitHub
https://github.com/NixOS/nixpkgs/pull/4796/files#r21892023.

[teh]

Apologies, I don't know. Is there a lot of contention on the test machine? Can you strace to figure out where it gets stuck if anywhere? perf may also hint at issues.

We've decided not to use gitlab in the end because it feels way too fragile :(

[offlinehacker]

@teh, yeah, it's kinda fragile, but still, thanks for the service. Can i
ask what are you using instead?

On Tue, Dec 16, 2014 at 3:46 PM, teh [email protected] wrote:

In nixos/modules/services/misc/gitlab.nix
#4796 (diff):

•  wantedBy = [ "multi-user.target" ];
  

•  environment.HOME = "${cfg.stateDir}/home";
  

•  environment.UNICORN_PATH = "${cfg.stateDir}/";
  

•  environment.GITLAB_PATH = "${pkgs.gitlab}/share/gitlab/";
  

•  environment.GITLAB_APPLICATION_LOG_PATH = "${cfg.stateDir}/log/application.log";
  

•  environment.GITLAB_SATELLITES_PATH = "${cfg.stateDir}/satellites";
  

•  environment.GITLAB_SHELL_PATH = "${pkgs.gitlab-shell}";
  

•  environment.GITLAB_REPOSITORIES_PATH = "${cfg.stateDir}/repositories";
  

•  environment.GITLAB_SHELL_HOOKS_PATH = "${cfg.stateDir}/shell/hooks";
  

•  environment.BUNDLE_GEMFILE = "${pkgs.gitlab}/share/gitlab/Gemfile";
  

•  environment.GITLAB_EMAIL_FROM = "${cfg.emailFrom}";
  

•  environment.GITLAB_SHELL_CONFIG_PATH = "${cfg.stateDir}/shell/config.yml";
  

•  environment.GITLAB_SHELL_SECRET_PATH = "${cfg.stateDir}/config/gitlab_shell_secret";
  

•  environment.GITLAB_HOST = "${cfg.host}";
  

•  environment.GITLAB_DATABASE_HOST = "${cfg.databaseHost}";
  

•  environment.GITLAB_DATABASE_PASSWORD = "${cfg.databasePassword}";
  

Apologies, I don't know. Is there a lot of contention on the test machine?
Can you strace to figure out where it gets stuck if anywhere? perf may also
hint at issues.

We've decided not to use gitlab in the end because it feels way too
fragile :(

—
Reply to this email directly or view it on GitHub
https://github.com/NixOS/nixpkgs/pull/4796/files#r21901594.

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.11 (GNU/Linux)

mQENBFEY1PEBCADPOfERF2wo4qeoq9L1m2z4pKfWqNd4B6BsoFUWPNd7BXmY+9JG
jJddSkmYobWec7XjAFTBL0Xbttt+rK9SIED2dCOmU1FYMQElhGlM3PNA3kaiQFeV
ijgH318GCfZzDd0dWa5TN/IshVeWXwgngsIEmZTVf1VSeb3eO3B8Fxe3zsSLUq0b
71MmU5eLVP9pMkm5V5BTYp+lV70FIekKygkKq+uTDo1csWUatbs4Qvgv37Bymy2t
oTwOBXGoinQk5N/6asR1jWs3vKv0L0SruoZy/kEG/jXb4l2OZUP85EVMganYKouE
OchVmcmhBdWV+t3HK4r2ATfyEcMRzvzSflA1ABEBAAG0Jkpha2EgSHVkb2tsaW4g
PGpha2FodWRva2xpbkBnbWFpbC5jb20+iQE+BBMBAgAoBQJRGNTxAhsDBQkB4TOA
BgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRD6Zxi5hZclKnXNCACLOKa8abQp
eTWv9SXUwC7LVM5pP2mXcgn+Ipqr6YWBdLx4Iij0YlvUok9VeKvwTpUlT+cx++o3
wCM3AYrUyJE+zrtw49lInUmutz9seqJLU895oq+D+UuGoORrLBpEZrYR5f83uUmQ
E3Z1ZmWrNGXYtITWDtVZD/KauMF2nkPcmy/XaYXhd4WHD81DGNlKtGAHig6A3Phc
8Mr0A4yLDeRQJm8lCFEsxMJUNTupgY+ybbsMfVGx1gQvvGOTioV8CLCoRchUCCcm
YPArFg40KzIDSNjwdo9EVZDnlPx1hbOppfQydxP+JVnZsqoYmVY4UhIWi/NfOl3V
UMjl338INW1zuQENBFEY1PEBCADRSIfelOMjaTH7IfpMFFUc5Gys//njFnW9QAUg
wyfs2AFxUp6vKQ7nxXQiJXVhKTwe9iqo+oGxaHp4AeTjC7vXsfMuF5g5lfttbAo3
YEobEe6OG5so41nbwan6SyeIIQ2AmQqJBw8TKKMSec2qUN0Pw7iZRs0o9uJM/obG
DPsAsMOQgNLxJyMCP7X2jBtDXxkMFVHMmk50Tl3h3Fi9qWuNxgTXjs0tUvKkXiu2
Pco952jnm7HpCIKBek2pqR/UJXXb5qxy5G6Lc0qaMWZ5GKnSMTJmTY6Xl44EnaLK
zh0rqgF9qpoWck470ZbiGASMtB008hy2l0cyxUfvDaS3tY4hABEBAAGJASUEGAEC
AA8FAlEY1PECGwwFCQHhM4AACgkQ+mcYuYWXJSoT6AgAkvzvC0EGmeCR3cn9O3Gf
yG00Kqk9/1gJvlphis7AAce8iUgU+4xd94Vp0u8rghpdy88xKN5lF1W2YZQmmBaf
AVe6b7TOg6kxc3GKkVsWDxNyQKkpB46BwefIGaSljH7502X9aEWosrqWyJJNYCtt
QDit4BysX0Ww3Ka5Rx6ZFhC9ybPKoW2i8JwpyBaXDt7R2k+PC/ClBf9qzL+sb2es
zh/zCMVKNdm8KUITHU/5lgn2qZpUFZwiASPCMGGFP9u8g6UKeUTYTPD+GWaHIW63
RAgNIAffxx0M1r3P/2ipkAdI3NX/1iBKDQNG8Odsf+BswFKrNCnyUDdLPvJAhODS
gw==
=tmrm
-----END PGP PUBLIC KEY BLOCK-----

[teh]

We're on github for now. We're also going to have a look at gogs (https://github.com/gogits/gogs) at some point.

For gogs I couldn't find anything on how they're updating their database on a version bump so I need to check the code. (for sentry I just run migrate before each start which takes a few seconds but that's fine).