VARNISH 4.0 ON CENTOS 6 WITH SELINUX


As of this writing, Varnish 4.0 won't run on a CentOS 6.x server when SELinux is enforcing.

There is probably nothing more common for a RedHat/CentOS admin to do than disabling SELinux. But we can't always disable SELinux.

Varnish won't start

Below you can see we have a CentOS 6.5 server and SELinux is enforcing.

[[email protected] sysconfig]# cat /etc/redhat-release
CentOS release 6.5 (Final)
[[email protected] sysconfig]# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   enforcing
Mode from config file:          error (Success)
Policy version:                 24
Policy from config file:        targeted

But varnish 4.0 can't be started.

[[email protected] sysconfig]# service varnish start
Starting Varnish Cache:                                    [FAILED]
[[email protected] sysconfig]# rpm -qa | grep varnish
varnish-4.0.2-1.el6.x86_64
varnish-libs-4.0.2-1.el6.x86_64

If I disable SELinux and reboot varnish will start.

There are a few error logs in /var/log/audit/audit.log, and I'll drop them in here just for google searches.

[[email protected] audit]# grep AVC /var/log/audit/audit.log
type=AVC msg=audit(1413316302.956:4): avc:  denied  { chown } for  pid=1050 comm="varnishd" capability=0  scontext=system_u:system_r:varnishd_t:s0 tcontext=system_u:system_r:varnishd_t:s0 tclass=capability
type=AVC msg=audit(1413316303.785:5): avc:  denied  { fowner } for  pid=1050 comm="varnishd" capability=3  scontext=system_u:system_r:varnishd_t:s0 tcontext=system_u:system_r:varnishd_t:s0 tclass=capability
type=AVC msg=audit(1413316303.785:5): avc:  denied  { fsetid } for  pid=1050 comm="varnishd" capability=4  scontext=system_u:system_r:varnishd_t:s0 tcontext=system_u:system_r:varnishd_t:s0 tclass=capability
type=AVC msg=audit(1413316832.823:52): avc:  denied  { chown } for  pid=1573 comm="varnishd" capability=0  scontext=unconfined_u:system_r:varnishd_t:s0 tcontext=unconfined_u:system_r:varnishd_t:s0 tclass=capability
type=AVC msg=audit(1413316881.454:59): avc:  denied  { chown } for  pid=1614 comm="varnishd" capability=0  scontext=unconfined_u:system_r:varnishd_t:s0 tcontext=unconfined_u:system_r:varnishd_t:s0 tclass=capability
type=AVC msg=audit(1413317112.992:65): avc:  denied  { chown } for  pid=1704 comm="varnishd" capability=0  scontext=unconfined_u:system_r:varnishd_t:s0 tcontext=unconfined_u:system_r:varnishd_t:s0 tclass=capability

There's a bug ticket for this problem and it was fixed in Fedora 19. But I'm not running Fedora 19, I'm running CentOS 6.5.

SELinux troubleshooting

I thought this might be a good time to learn about SELinux. For the last few years I've been working at a place that is almost exclusively an Ubuntu shop. Before that, nowhere actually used SELinux, they all just turned it off. So, I have never worked at an organization that left SELinux on. Time to change that!

There's a good document here that I'm following to create a policy that will allow varnish 4 to startup and work properly.

First let's install some packages.

[[email protected] ~]$ sudo yum install -y policycoreutils-python setroubleshoot-server

Now we can translate the audit log into something human readable.

[[email protected] ~]# sealert -a /var/log/audit/audit.log > audit.txt

There's a few alerts in there, but they basically look like this:

found 3 alerts in /var/log/audit/audit.log
--------------------------------------------------------------------------------

SELinux is preventing /usr/sbin/varnishd from using the chown capability.

*****  Plugin catchall (100. confidence) suggests  ***************************

If you believe that varnishd should have the chown capability by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# grep varnishd /var/log/audit/audit.log | audit2allow -M mypol
# semodule -i mypol.pp

So I ran this:

[[email protected] ~]# grep varnishd /var/log/audit/audit.log | audit2allow -M varnishpol

Which created a couple of files, one data file and one configuration file. Let's look at the configuration file.

[[email protected] ~]# cat varnishpol.te

module varnishpol 1.0;

require {
    type varnishd_t;
    class capability { fowner chown fsetid };
}

#============= varnishd_t ==============
allow varnishd_t self:capability { fowner chown fsetid };

As can be seen above, varnishd_t is getting three new capabilities: fowner, chown, fsetid.

Using the varnishpol.pp file I can load the varnish policy.

[[email protected] ~]# semodule -i varnishpol.pp

And now start varnish, this time with SELinux enforcing.

[[email protected] ~]# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   enforcing
Mode from config file:          error (Success)
Policy version:                 24
Policy from config file:        targeted
[[email protected] ~]# service varnish start
Starting Varnish Cache:                                    [  OK  ]
[[email protected] ~]# ps ax | grep varnish
 2319 ?        SLs    0:00 /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 120 -u varnish -g varnish -S /etc/varnish/secret -s malloc,70M
 2321 ?        Sl     0:00 /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 120 -u varnish -g varnish -S /etc/varnish/secret -s malloc,70M

The module was installed in /etc/selinux/targeted/modules/active/modules .

[[email protected] modules]# pwd
/etc/selinux/targeted/modules/active/modules
[[email protected] modules]# ls varnish*
varnishd.pp  varnishpol.pp

It's alongside the existing varnishd.pp module, which is also found bzipped in:

[[email protected] modules]# locate varnishd.pp
/etc/selinux/targeted/modules/active/modules/varnishd.pp
/usr/share/selinux/targeted/varnishd.pp.bz2

So the bzipped file is from the RPM, but what is in /etc/selinux is more like a configuration file, which makes sense.

[[email protected] modules]# rpm -qf /usr/share/selinux/targeted/varnishd.pp.bz2
selinux-policy-targeted-3.7.19-231.el6_5.3.noarch
[[email protected] modules]# rpm -qf varnishd.pp
file /etc/selinux/targeted/modules/active/modules/varnishd.pp is not owned by any package

I believe that once that file is in /etc/selinux... it will be installed on a reboot.

Conclusion

In the end, by granting varnishd some additional capabilities via SELinux it is allowed to start. However, I still have several questions that I will look to answer in future blog posts:

  1. How do the pp files end up getting unzipped and landed in /etc/selinux...?
  2. Are the additional capabilities we gave varnishd correct? Is it too much?
  3. What's the proper way to add the selinux module? RPM it and install it in the /usr/share/selinux dir?
  4. Should I grab the pp file from Fedora 19 (or later) and use that?
  5. How does that policy know what binary is what? I don't see anything showing which varnishd it is supposed to be allowing.
  6. What's the difference between apparmor and SELinux? Why don't I hear of Ubuntu admins turning off apparmor as much as I do turning off SELinux?

So, still lots of questions, but so far a working varnishd with SELinux still enabled.