Gerrit as a systemd service unit…

… and other install gotchas

6 Apr '16

TL;DR: Click for the example service unit configuration file.

I love Gerrit for code review at $work (Not my current work mind - personal opinion and all). GitHub and GitLab are great tools (I’m sure I’ve missed others). The pull request (PR) workflow is more suited to contributors with varying backgrounds, but Gerrit’s workflow feels faster. I think this is because a PR can be multiple commits, whereas in Gerrit a changeset is roughly equivalent to a commit (once merged), plus the changeset contains all revisions as patchsets. Alice reviews a patchset, Bob amends the commit, pushes it and Alice can see both patchsets, and crucially the diff of both patchsets also. I’m not saying other tools don’t do this, but Gerrit worked for our team.

Deployment however is a bit of a hassle. Oh, and the interface is ugly.

Note: This is for Gerrit v1, not v2 which was recently released.

Short intro

Gerrit is a Java program. I’ll be comparing it to Go CD, because I’m messing about deploying that at the moment, but a fairer comparison would probably be Jenkins.

As far as Java programs go, it’s pretty light weight, certainly compared to Go CD or Eclipse. Although more users will quickly eat into resources such as file descriptors - to be expected!

Go CD has packages for different platforms (and yum/apt repositories), and installs itself in fairly standard locations, plus the documentation is top notch.

Gerrit comes as a .war file, so you’ll have to check the documentation to get it deployed. Gerrit’s documentation isn’t great, and whatever you do, don’t click the Wiki link. Very confusing.

Deploying Gerrit

Turns out you have to “initialise a site” via:

java -jar gerrit.war init -d "<site>"

This creates a directory structure similar to the Filesystem Hierarchy Standard under <site>, including bin, cache, data, etc, lib, logs, and tmp. Not what I want as a sysadmin. FYI, the PID file lives in logs.

To be fair, some of this is tweak-able, e.g. the location of git repositories can be configured via gerrit.basePath. However, you have to do this before running init, otherwise the default repositories All-Projects.git and All-Users.git aren’t created. Also, make sure the folder has the correct permissions - I’ve had instances where Gerrit hasn’t set up the default projects and shown no errors. Re-running init will not create these default repositories if the site already exists.

Also keep in mind that you need to run the reindex operation every time you modify the Gerrit config file - and for this, the Gerrit instance can’t be running.

Webserver woes

Other gotchas include that the built-in Java webserver Jetty is hard to set up for SSL. In this day and age that should be unacceptable, although luckily the solution is also straight-forward: Run Gerrit behind Nginx. At that point, I’d rather have Nginx do the authentication - if you know how to do this, please get in touch.

From init.d script to systemd service unit

The script to manage Gerrit is a) huge, b) hack-y, and c) an init.d script. I wanted a systemd script, because it’s usually simpler and even though systemd does support init.d scripts, bad things can happen if you interact via the script itself. With gerrit, this is particularly bad, as the script doesn’t live in /etc/init.d/, but in <site>/bin/gerrit.sh.

This gerrit.sh script actually reads from /etc/default/gerritcodereview, environmental variables, and the gerrit config file, and debugging this is a pain.

Instead, the systemd service unit configuration file (service file) lets us set many of these options ourself, and makes it much easier to support multiple sites (although why you would do this on the same machine/VM/container is beyond me, as Gerrit has fully featured auth and permissions). The options systemd inherits or replaces are:

i.e. these are set in the service unit config, and not in the Gerrit config with this approach (unlike the init.d script which actually reads the config file). I think that’s all of them.

The gerrit.sh script also has a section called “Configure sane ulimits for a daemon our size”, which as a sysadmin makes me worried. This isn’t a good thing to happen automatically. But again, systemd makes it easy to set the ulimits for Gerrit if you wish to do so.

In the end, the systemd service unit configuration file is about 20-30 lines, instead of 565 for a hack-y script. Granted, the service file is less… general? But well worth it for consistency.

The service unit configuration, in all its glory:

[Unit]
Description=Web based code review and project management for Git based projects
After=syslog.target network.target remote-fs.target

[Service]
Type=simple
User={{ gerrit_user }}
ExecStart=/usr/bin/java -DGerritCodeReview=1 -jar {{ gerrit_home }}/bin/gerrit.war daemon -d {{ gerrit_home }}
ExecStop=/bin/kill -s SIGINT $MAINPID
# stupid java exit codes
SuccessExitStatus=130 SIGINT
SuccessExitStatus=143 SIGTERM

# default gerrit ulimits set by the init.d script... your choice
LimitCORE=0
LimitDATA=unlimited
LimitFSIZE=unlimited
LimitRSS=unlimited
# NOFILE :  GERRIT_FDS, determined by "core.packedGitOpenFiles" in the script
LimitNOFILE=1024
LimitCPU=unlimited
LimitAS=unlimited
LimitLOCKS=unlimited

[Install]
WantedBy=multi-user.target

sysadmin, ansible, Linux

Newer Older