Blocking Rootkits

The variety of malicious computer software keeps expanding, and the bad guys, at least so far, have shown no flagging of their ingenuity.  One of the nastiest kinds of current malware is the so-called “root kit”, which can be built to work by itself, or to act as a “host” for other malicious programs.  The name is somewhat misleading, since this kind of software does not necessarily give system administrator (= ‘root’, in Unix and Linux) access (it may, however, provide that function once it is installed).  Its key characteristic is that it works at the operating system level, and attempts to subvert OS functions in order to conceal its own presence.  For example, it might intercept system calls to examine its own process identifier and return a “no such process” indication, and intercept file system requests to make itself “invisible” by normal user functions.  The ideas that Ken Thompson outlined in his 1983 ACM Turing Award lecture, “Reflections on Trusting Trust”, such as subverting the C compiler or the “login” command are similar to rootkit techniques.   But whereas Thompson’s backdoors were inserted into system software when it was built, a rootkit inserts itself into the running system, by modifying OS data structures.

Now a group of researchers at North Carolina State University have published a paper [PDF] that suggests a new and potentially promising way of protecting against rootkits.  They observe that, in many cases, important OS control data is located in a variety of places within the OS kernel’s privileged address space.  This makes protecting all the data somewhat tricky, because hardware protection mechanisms (the most reliable kind) typically work only at the level of single data pages.  Their approach is interesting, and even to some extent elegant.  They provide a small, special purpose virtual machine environment (a hypervisor) in which the operating system runs as a guest.  The hypervisor, HookSafe, collects all the sensitive kernel data structures and relocates them in one or more contiguous pages of the virtual machine’s address space.  The hardware write-protection mechanism used by the hypervisor can then be used to allow free reading of the data, but to forbid writing.

One key observation behind our approach is that a kernel hook, once initialized, may be frequently “read”-accessed, but rarely “write”-accessed. As such, we can relocate those kernel hooks to a dedicated page-aligned memory space and then regulate accesses to them with hardware-based page-level protection.

The team claims that, in initial tests of the system with a Linux OS and nine real-world rootkits, they blocked all of the attacks, with only minimal performance overhead.  The paper is due to be presented at the 16th ACM Conference on Computer and Communications Security, which begins tomorrow in Chicago.