Security in Depth for Linux Software

Chris Evans and myself have presented last week at Hack In The Box Malaysia about "Security in Depth for Linux software". You can find the slides here.

The talk was focused on writing good code and sandboxing.

The writing goode code part was using vsftpd as an example, since Chris has got this right for ten years now.

In the second part, we defined sandboxing, which we also call discretionary privilege dropping, as the ability to drop privileges programmatically and without administrative authority on the machine.

We explained some of the conceptual differences between sandboxing in this sense, where the application writer chooses to make part of his code run without certain privileges, and Mandatory Access Control systems, where the application itself doesn't make the policy.

From an application writer perspective, sandboxing facilities are desirable since they will allow your code to run with lower privileges on all machines. On the other hand, MAC is desirable from a system administrator or distribution maintainer perspective as it will allow one policy to rule over many applications and to enforce certain security properties on the system.

While Linux has a fair number of MAC systems available, sandboxing options are for now very limited. There is some hope that the ftrace framework or SELinux bounded types may allow this in the future (see also Adam Langley's post on LSMSB), but this will not be widely available anytime soon.

We demonstrated different ways of overcoming those limitations on readily available Linux kernels, focusing on three designs experimented or used in vsftpd and Chromium.

• Using ptrace(), vsftpd experiment
  
• The setuid sandbox design (Julien Tinnes, Tavis Ormandy), Chromium
  
• The SECCOMP sandbox design (Markus Gutschke, Adam Langley), Chromium
  

CVE-2009-2698: udp_sendmsg() vulnerability

EDIT: p0c73n1 has posted an exploit for this to milw0rm as did andi@void.at, and spender wrote "the rebel"

Tavis Ormandy and myself have recently reported CVE-2009-2698 which has been disclosed at the beginning of the week.

This flaw affects at least Linux 2.6 with a version < 2.6.19.

When we ran into this, we realized the newest kernel versions were not affected by the PoC code we had. The reason for this was that Herbert Xu had found and corrected a closely related bug. Linux distributions running on 2.6.18 and earlier kernels did not realize the security impact of this fix and did not backport it.
This is a good example on how hard it is to backport relevant fixes to maintained stable versions of the kernel.

If you look at udp_sendmsg, you will see that the rt routing table is initialized as NULL and some code paths can lead to call ip_append_data with a NULL rt. ip_append_data() obviously doesn't handle this case properly and will cause a NULL pointer dereference.

Note that this is a data NULL pointer dereference and mapping code at page zero will not lead to immediate privileged code execution for a local attacker. However, controlling the rtable structure seems to give enough control to the attacker to elevate privileges.

Since it's hard to guarantee that ip_append_data will never be called with a NULL *rtp, we believe that this function should be made more robust by using this patch.

Here's one way to trigger this vulnerability locally:

$ cat croissant.c
#include <sys/types.h>
#include <sys/socket.h>
#include <string.h>

int main(int argc, char **argv)
{
int fd = socket(PF_INET, SOCK_DGRAM, 0);
char buf[1024] = {0};
struct sockaddr to = {
.sa_family = AF_UNSPEC,
.sa_data = "TavisIsAwesome",
};

sendto(fd, buf, 1024, MSG_PROXY | MSG_MORE, &to, sizeof(to));
sendto(fd, buf, 1024, 0, &to, sizeof(to));

return 0;
}

An effective implementation of mmap_min_addr or the UDEREF feature of PaX/GrSecurity would prevent local privilege escalation through this issue.

Linux NULL pointer dereference due to incorrect proto_ops initializations (CVE-2009-2692)

EDIT2: Here is RedHat's official mitigation recommendation
EDIT3: Brad Spengler also wrote an exploit for this and published it. The bug triggering is based on our exploit which leaked to Brad though the private vendor-sec mailing list. He implements the personality trick Tavis and I published in June to bypass mmap_min_addr and also makes use of a feature that allows any unconfined user to gain the right to map at address zero in Redhat's default SELinux policy. He wrote a reliable shellcode for this one that should work pretty much anywhere on x86 and x86_64 machines.
EDIT4: if you use Debian or Ubuntu on your machine, I have specifically updated the kernelsec Debian/Ubuntu GrSecurity packages to protect against this bug and others.
EDIT5: Zinx wrote an ARM Android root exploit
EDIT6: Ramon de Carvalho Valle wrote a PPC/PPC64/x86_64/i386 exploit

Tavis Ormandy and myself have recently found and investigated a Linux kernel vulnerability (CVE-2009-2692). It affects all 2.4 and 2.6 kernels since 2001 on all architectures. We believe this is the public vulnerability affecting the greatest number of kernel versions.

The issue lies in how Linux deals with unavailable operations for some protocols. sock_sendpage and others don't check for NULL pointers before dereferencing operations in the ops structure. Instead the kernel relies on correct initialization of those proto_ops structures with stubs (such as sock_no_sendpage) instead of NULL pointers.

At first sight, the code in af_ipx.c looks correct and seems to initialize .sendpage properly. However, due to a bug in the SOCKOPS_WRAP macro, sock_sendpage will not be initialized. This code is very fragile and there are many other protocols where proto_ops are not correctly initialized at all (vulnerable even without the bug in SOCKOPS_WRAP), see bluetooth for instance.

So it was decided that instead of patching all those protocols and continue to rely on this very fragile code, sock_sendpage would get patched to check against NULL. This was already the way sock_splice_read and others were handled.

Since it leads to the kernel executing code at NULL, the vulnerability is as trivial as it can get to exploit (edit: that's for local privilege escalation and on Intel architectures): an attacker can just put code in the first page that will get executed with kernel privileges. Our exploit took a few minutes to adapt from a previous one:

$ ./leeches
// ------------------------------------------------------

// sendpage linux local ring0
// ---------------- taviso@sdf.lonestar.org, julien@cr0.org
// leeches.c:Aug 11 2009
// GreetZ: LiquidK, lcamtuf, Spoonm, novocainated, asiraP, ScaryBeasts, spender, pipacs, stealth, jagger, redpig, Neel and all the other leeches we forgot to mention!
Enjoy some photography while at ring0 @ http://flickr.com/meder
For our webapp friends, here is an XSS executing at ring 0: javascript:alert(1);
shellcode now executing chmod("/bin/sh", 04755), welcome to ring0
Killed
$ sh
# id
uid=1000(julien) gid=1000(julien) euid=0(root)

On x86/x86_64, this issue could be mitigated by three things:

• the recent mmap_min_addr feature. Note that this feature has known issues until at least 2.6.30.2. See also this LWN article.
• on IA32 with PaX/GrSecurity, the KERNEXEC feature (x86 only)
• not implementing affected protocols (a.k.a., reducing your attack surface by disabling what you don't need): PF_APPLETALK, PF_IPX, PF_IRDA, PF_X25, PF_AX25, PF_BLUETOOTH, PF_IUCV, IPPROTO_SCTP/PF_INET6, PF_PPPOX, PF_ISDN, but there may be more. (Update: See RedHat's mitigation)

This patch should be applied to fix this issue.

You can read our advisory here.

Note: this has been featured on Slashdot, OSNews, TheRegister, ZDNet and others

Old school local root vulnerability in pulseaudio (CVE-2009-1894)

Today was chosen as disclosure day for CVE-2009-1894.

Tavis Ormandy and myself have recently used the fact that pulseaudio was set-uid root to bypass Linux' NULL pointer dereference prevention. This technique is relying on a limitation in the Linux kernel and not on a bug in pulseaudio. But we also found one unrelated bug in pulseaudio.

Since it's set-uid root, we thought we would give pulseaudio a quick look. In the very first lines of main(), you can find the following:

if (!getenv("LD_BIND_NOW")) {
char *rp;

putenv(pa_xstrdup("LD_BIND_NOW=1"));
pa_assert_se(rp = pa_readlink("/proc/self/exe"));
pa_assert_se(execv(rp, argv) == 0);
}
So, pulseaudio is re-executing itself through /proc/self/exe, so that the dynamic linker performs all relocation immediately at load-time.

There is an obvious race condition here. /proc/self/exe is a symbolic link to the actual pathname of the executed command: by creating a hard link to /usr/bin/pulseaudio, we control this pathname, and consequently the file under this pathname. Knowing this, the exploitation is trivial (Note that rename() is atomic, or alternatively note how __d_path() works with deleted entries).

It's also interesting to note that any operation performed on /proc/self/exe is guaranteed by the kernel to be performed on the same inode than the one that got executed (see proc_exe_link), something that two of my colleagues have recently pointed out to me. So if they had re-executed themselves by using /proc/self/exe directly, without going through readlink() first, they would not have been vulnerable. And actually they weren't before, if you read the Changelog, you'll find:

2007-10-29 15:33 lennart * : use real path of binary instead of /proc/self/exe to execute ourselves

Oops! (Thanks to my colleague Mike Mammarella for digging this)

Like the vulnerability in udevd, this is a very good example of a non memory corruption vulnerability which is trivial to exploit very reliably and in a cross-architecture way.

So, why does pulseaudio have the set-uid bit set, you may ask ? For real-time performances reasons it wants to keep CAP_SYS_NICE but will drop all other privileges.

This vulnerability could have been avoided if the principle of least privilege had been followed: Since privileges are not required to re-exec yourself, dropping privileges should have been the first thing pulseaudio did. Here it's only the second thing it does, and it was enough to make most Linux Desktops vulnerable.

If your distribution of choice did not patch this yet, or if you want to reduce your attack surface, you're advised to chmod u-s /usr/bin/pulseaudio. Also note that as with every setuid binary update, you should also check that your users didn't create "backup" vulnerable copies (hardlink), waiting to own your box with known vulnerabilities while you think you are safe from those.

PS: Here are two brain teasers for you:

1. Find a cool way to perform an action after execve() has succeeded in another process, but before main() executes. First, I've used a FD_CLOEXEC read descriptor in a pipe and a SIGPIPE handler, but while it gives good results in practice, there is not guarantee as to when the signal will get delivered. I've finally found (with a hint from Tavis) a 100% reliable way to do it that is always guaranteed to work at first try. Of course, such a level of sophistication is absolutely not needed for this exploit.

2. Since pulseaudio allows you to load arbitrary libraries, it allows you to run arbitrary code with CAP_SYS_NICE as a feature. In the light of NUMA coming to the desktop through QPI, can you do something more interesting than what you would first expect with this?

Bypassing Linux' NULL pointer dereference exploit prevention (mmap_min_addr)

EDIT3: Slashdot, the SANS Institute, Threatpost and others have a story about an exploit by Bradley Spengler which uses our technique to exploit a null pointer dereference in the Linux kernel.
EDIT2: As of July 13th 2009, the Linux kernel integrates our patch (2.6.31-rc3). Our patch also made it into -stable.
EDIT1: This is now referenced as a vulnerability and tracked as CVE-2009-1895

NULL pointers dereferences are a common security issue in the Linux kernel.

In the realm of userland applications, exploiting them usually requires being able to somehow control the target's allocations until you get page zero mapped, and this can be very hard.

In the paradigm of locally exploiting the Linux kernel however, nothing (before Linux 2.6.23) prevented you from mapping page zero with mmap() and crafting it to suit your needs before triggering the bug in your process' context. Since the kernel's data and code segment both have a base of zero, a null pointer dereference would make the kernel access page zero, a page filled with bytes in your control. Easy.

This used to not be the case, back in Linux 2.0 when the kernel's data segment's base was above PAGE_OFFSET and the kernel had to explicitely use a segment override (with the fs selector) to access data in userland. The same rough idea is now used in PaX/GRSecurity's UDEREF to prevent exploitation of "unexpected to userland kernel accesses" (it actually makes use of an expand down segment instead of a PAGE_OFFSET segment base, but that's a detail).

Kernel developpers tried to solve this issue too, but without resorting to segmentation (which is considered deprecated and is mostly not available on x86_64) and in a portable (cross architectures) way. In 2.6.23, they introduced a new sysctl, called vm.mmap_min_addr, that defines the minimum address that you can request a mapping at. Of course, this doesn't solve the complete issue of "to userland pointer dereferences" and it also breaks the somewhat useful feature of being able to map the first pages (this breaks Dosemu for instance), but in practice this has been effective enough to make exploitation of many vulnerabilities harder or impossible.

Recently, Tavis Ormandy and myself had to exploit such a condition in the Linux kernel. We investigated a few ideas, such as:

• using brk()
• creating a MAP_GROWSDOWN mapping just above the forbidden region (usually 64K) and segfaulting the last page of the forbidden region
• obscure system calls such as remap_file_pages
• putting memory pressure in the address space to let the kernel allocate in this region
• using the MAP_PAGE_ZERO personality

All of them without any luck at first. The LSM hook responsible for this security check was correctly called every time.

So what does the default security module do in cap_file_mmap? This is the relevant code (in security/capability.c on recent versions of the Linux kernel):

if ((addr < mmap_min_addr)
&& !capable(CAP_SYS_RAWIO))
  return -EACCES;
return 0;

Meaning that a process with CAP_SYS_RAWIO can bypass this check. How can we get our process to have this capability ? By executing a setuid binary of course! So we set the MMAP_PAGE_ZERO personality and execute a setuid binary. Page zero will get mapped, but the setuid binary is executing and we don't have control anymore.
So, how do we get control back ? Using something such as "/bin/su our_user_name" could be tempting, but while this would indeed give us control back, su will drop privileges before giving us control back (it'd be a vulnerability otherwise!), so the Linux kernel will make exec fail in the cap_file_mmap check (due to the MMAP_PAGE_ZERO personality).

So what we need is a setuid binary that will give us control back without going through exec. We found such a setuid binary that is installed on many Desktop Linux machines by default: pulseaudio. pulseaudio will drop privileges and let you specify a library to load though its -L argument. Exactly what we needed!

Once we have one page mapped in the forbidden area, it's game over. Nothing will prevent us from using mremap to grow the area and mprotect to change our access rights to PROT_READ|PROT_WRITE|PROT_EXEC. So this completely bypasses the Linux kernel's protection.

Note that apart from this problem, the mere fact that MMAP_PAGE_ZERO is not in the PER_CLEAR_ON_SETID mask and thus is allowed when executing setuid binaries can be a security issue: being able to map page zero in a process with euid=0, even without controlling its content could be useful when exploiting a null pointer vulnerability in a setuid application.

We believe that the correct fix for this issue is to add MMAP_PAGE_ZERO to the PER_CLEAR_ON_SETID mask.
PS: Thanks to Robert Swiecki for some help while investigating this.

Time-stamp counter disabling oddities in the Linux kernel

The time-stamp counter (TSC) is part of the performance monitoring facilities provided on Intel processors. It's stored in a 64-bits MSR. Except for 64-bit wraparound (and of course reset), the TSC is guaranteed to be monotonically increasing by Intel, but not necessarily at a constant rate.
Historically, the TSC increased with every internal processor clock cycle, but now the rate is usually constant (even if the processor changes frequency) and usually equals the maximum processor frequency.

There are multiple ways of reading the value of the TSC MSR, a popular one is the RDTSC instruction. This instruction will load the value into EDX:EAX and is not privileged unless the Time-Stamp Disable (TSD) bit is set in CR4. Most operating systems will not set CR4.TSD on any thread, so programmers are free to use RDTSC in their Ring3 code.

The problem is that the TSC has been used as a tool in the past to mount side channel attacks. Two examples are "Cache Attacks and Countermeasures: the Case of AES" by Osvik, Shamir and Tromer and "Cache missing for fun and profit" by Colin Percival. (Less importantly, it has also been used to create exploits against race conditions in the Linux kernel such as this one)

In an attempt to kill RDTSC as a tool to conduct various mischiefs, Andrea Arcangeli, author of the SECCOMP prctl (that allows a thread to enter a sandboxed "computing mode" where only read, write, exit and sigreturn syscalls are allowed) tried to disable RDTSC by setting CR4.TSD in any thread that runs under seccomp (in 2.6.12).

That's where the oddities begin: I was recently surprised to see that a process I ran under seccomp had actually access to rdtsc. A quick look at the source code of my kernel revealed this:

#ifdef TIF_NOTSC
disable_TSC();
#endif

Note that TIF_NOTSC is not a config option! So I took a look at both thread_info_64.h and thread_info_32.h to discover that in the 64 bits version, TIF_NOTSC was not defined. As a consequence, on a 32 bits kernel, seccomp will disable the TSC in seccomp threads but will not on a 64 bits kernel (even for 32 bits processes). Chris Evans blogged previously about how a seemingly simple security technology such as seccomp could still have bugs. "Here's another one" I thought.

While tracking this bug, I found out that it wasn't a bug but a conscious decision by Andi Kleen to not disable TSC, but only on x86_64 (patch applied in 2.6.14) for performance reasons. I consider this a really odd decision: seccomp behaving differently on 64 bits and 32 bits kernels is a non sense! If you consider TSC disabling a security feature, it has to behave consistently or you should just remove it altogether. Here is a thread, started in November 2005 by Andrea Arcangeli who also regretted the lack of consistency.

But then, in Linux 2.6.23, this feature became impact-free, performance-wise. So at that point, I really consider not having it on x86_64 kernels a bug, not only a strange decision. As I mentioned previously, the bug is due to TIF_NOTSC not being defined for 64 bits kernels.
I wondered if this bug would still be there in recent Linux kernels despite the ongoing i386 and x86_64 merge. It wasn't in 2.6.27 where thread_info_64.h and thread_info_32.h have been merged into one thread_info.h file. But in fact, it was already corrected in 2.6.26 at the same time as a new feature, prctl(PR_SET_TSC), was introduced.

PR_SET_TSC lets you control the CR4.TSD flag for your thread: you can make your thread SIGSEGV on rdtsc. And this feature is another big oddity for me: if you consider rdtsc harmful, it would make sense to let a process drop the privilege to use RDTSC, but the weird thing here is that they don't forbid you to call prctl(PR_SET_TSC) again to clear the TSD flag and restore your privilege to use rdtsc! So I can't imagine what this is for, the only use case I can see would be in a ptrace sandbox.

Another use case would have been SECCOMP of course. By removing the automatic TSC disabling from seccomp, a thread could use PR_SET_TSC prior to using PR_SETCOMP if it wanted to disable rdtsc, thus making this behavior configurable. Since a thread under seccomp cannot call prctl(), the thread wouldn't have been able to re-enable it. But the problem would have been that existing code relying on SECCOMP might be expecting to drop TSC access without having to use PR_SET_TSC. But wait! This feature had never worked in the first place, it was the perfect time to change the behavior and finally fix this bug. Another oddity!

I should also discuss the whole idea of forbidding access to the useful rdtsc instruction in the first place. Could an attacker emulate this with a thread on another processor incrementing a counter manually anyway? Are the RTC, HPET or the gtod_data counter in the vsyscall page useable? How realistic are those side channel attacks in the first place? If they are, when could it become the easiest attack you can perform on a system ? That will be for another post.

Local bypass of Linux ASLR through /proc information leaks

EDIT2: Thanks to the efforts of Jake Edge who noticed our presentation, /proc/pid/stat information leak is now at least partially patched in mainline kernel, since 2.6.27.23
EDIT1: This is featured in an LWN article by Jake Edge

Tavis Ormandy and myself talked about locally bypassing address space layout randomization (ASLR) in Linux in a lightning talk at CanSecWest.

From Linux 2.6.12 to Linux 2.6.21, you could completely bypass ASLR when targeting local processes by reading /proc/pid/maps. Since Linux 2.6.22, if you cannot ptrace "pid", then you will see an empty /proc/pid/maps.

It has been known for at least 7 years now that /proc/pid/stat and /proc/pid/wchan could also leak sensitive information. Reading this information has been prevented in GRSecurity since the beginning as well as in this patch.

The question was: could you exploit this information to bypass ASLR in practice?
If you want to find out, it's easy: we've just published the slides and Tavis' tool!

Interesting vulnerability in udevd

I used to love exploiting memory corruption vulnerabilities. It usually requires some reverse engineering, good knowledge of the underlaying operating system and some ingenuity to write reliable exploits. And if you try to circumvent clever protections such as PaX, it can get very tricky.

But besides kernel vulnerabilities, exploitable memory corruption vulnerabilities these days are mostly buffer overflows. It's a bit monotonous.

I get more excited by other kind of vulnerabilities such as Solaris' telnet -froot or the Debian/OpenSSL fiasco.

Last night, my friend Raph pointed me to this udev flaw. If you read this patch you can notice an extra check in get_netlink_msg():
if ((snl.nl_groups != 1) || (snl.nl_pid != 0))

This checks if the message recieved by udevd had been sent to a specific multicast group (sending to netlink multicast groups is privileged and can only be done with CAP_NET_ADMIN) and also if it was sent from the kernel's unicast address.

From now on, the vulnerability is pretty obvious: before the patch, udevd didn't check the origin of messages it was recieving through netlink.

So can we spoof the kernel and send arbitrary messages to udevd? Yes! And it's easy, it suffices to create a NETLINK socket with the NETLINK_KOBJECT_UEVENT protocol and to send a unicast message to the correct unicast address. In udevd, this address will be the pid of the process who bound the NETLINK socket (udevd's parent). You can easily find it in /proc/net/netlink (thanks Phil)). Et voilà!

My idea to exploit this was to create a 666 device node that would give direct access to a mounted partition and to chmod +s some binary file we control by directly writing to the block device (there are userland tools and lib to do this easily, see debugfs for instance).

Phil also came-up with the idea of replacing /dev/urandom and /dev/random with /dev/zero (so called "debian emulation" backdoor).
Raph then found an even better way: on Ubuntu, Debian and others, you can exploit "95-udev-late.rules" and run arbitrary commands by using the "remove" action.

And that's it for a slick exploit. 40 lines of C (5 lines of Python for Phil). Pretty simple, cross architecture, reliable. And it can escape chroots and some MAC-constrained environments (as long as you can create netlink sockets).