Nuclear Elephant: Chrooting daemons and system processes HOW-TO

Security Papers

Chrooting daemons and system processes HOW-TO
Last Update: Wednesday, March 12 2003 13:30
Please notify me of any corrections or suggested additions

Jonathan A. Zdziarski
jonathan@nuclearelephant.com

Table of Contents
Part I: Introduction to Chrooting

1.1 What is chrooting?
1.2 When is it appropriate to chroot?
1.3 Can all daemons be chrooted?
1.4 Will chrooting affect my users?
1.5 What is entailed in chrooting?

Part II: Requirements gathering

2.1 Can we chroot this daemon?
2.2 An Introduction to truss, lsof, and ldd
2.3 Determining datafile dependencies
2.4 Creating a datafile strategy
2.5 Determining library dependencies
2.6 Finding a good home for the jail

Part III: Setting up a chrooted environment

3.1 Creating an empty jail
3.2 Copying program and datafiles, configuring cron
3.3 Copying libraries
3.4 Creating devices
3.5 Altering startup scripts
3.6 The Final Product
3.7 Logging through syslog

Part IV: Advanced Topics

4.1 How to tell if you're chrooted
4.2 Breaking out of a chroot()ed cell
4.3 Frequent chroot() mistakes

Part I

1.1 What is chrooting?
The command/function chroot is short for 'change root', and is designed to change the filesystem root for the environment it is applied to. This means the initial slash (/) in any path names are made relative to the chrooted path. For example, if a file called: /home/jonz/hello.txt exists on the system, and then I chrooted to /home/jonz, the file would then exist, in my chrooted environment, as: /hello.txt

The purpose of chrooting is designed to create an impenetrable (theoretically) "jail" protecting what is being chrooted from being able to read or modify any files outside of the chrooted environment. In the example above, I would be unable to access any files outside of /home/jonz, since / is now pointing to /home/jonz. Chrooting is commonly used to jail users in multiuser environments to protect system files. Chrooting can also be used to jail system daemons to help prevent them from being viable targets for hackers. If a hacker should exploit a vulnerability in a chrooted system daemon, their ability to affect files outside of the jail, or obtain a root shell is significantly more difficult. One big reason for this is that a shell is no longer part of the environment's path, so even if the hacker blows the stack away there's no shell to drop to. Many people have claimed to be able to break out of a chrooted jail, but in many cases it was from a shell (which doesn't exist in our case). Breaking out of a daemon-environment jail is at the very least, extremely difficult.

1.2 When is it appropriate to chroot daemons or system processes?
Chrooting daemons is a practical method of adding an additional layer of security to your system. Many system processes and third party applications already have some safeguards against vulnerability exploits. Many tools now have the ability to run as non-root users, which makes it harder for hackers to attack root.

Network security layers such as firewalls, TCP wrappers, filters, and etcetera also add to the overall security of a system. Like all of these, chrooting is appropriate for most implementations, where it is possible to do so without compromising functionality.

1.3 Can all daemons be chrooted?
Technically you can chroot anything you like including your mother's casserole, but in some cases chrooting is not always possible without "breaking" something, or in other circumstances, without an elaborate nonconventional configuration not worth the trouble . Some daemons cannot function properly in a chrooted environment due to the complexity of their functions. For example, sendmail must have the ability to access users' home directories to search for .forward files. There is no practical way to chroot sendmail without creating an elaborate, time consuming "mirror". This is why sendmail has an alternative solution, smrsh (sendmail restricted shell). A majority of system daemons, however, can be safely chrooted with little effort.

1.4 Will chrooting affect my users?
If it is done correctly, your users should notice no difference in system behavior. Chrooting in itself will not directly affect your users or alter your operating system. The existing system is commonly left untouched, while small "jails" are created to provide services from. Keep in mind also, we're not talking about taking advantage of chroot packages existing in ftp or ssh daemons. That is a similar, yet different concept than the one we're discussing here. We're discussing chrooting the system daemons that provide these services to transparently be able to get their job done with additional security.

1.5 What is entailed in chrooting?
Chrooting any daemon is a process and commonly involves the following steps to producing a viable chrooted daemon:

Creating a "jail" directory
Copying the daemon and any required datafiles
Copying required system libraries
Altering any startup scripts to launch the daemon in the new environment

Steps 2 and 3 commonly consume most time. Step 2, in particular, requires a certain level of creativity in some circumstances. For example, a mail popper has to have access to the mail folders on the system. Copying the mail folders to the jail will not produce the desired results, so we then must create a jail "around" the mail folders. We'll get into the complexities of this later.

Part II

2.1 Can we chroot this daemon?
Before even creating a directory or copying a single file, we need to ask ourselves some important questions. The first is whether or not the daemon we're looking at can be chrooted. We're going to be working with three examples of system daemons below, and pose the same key questions:

Does the daemon access files in one general location?

Is it viable to copy the files either once, or periodically using cron?

Is it prudent to create a jail "around" a specific set of files?

If you answered 'No' to all three of these questions, it may not be a good idea to chroot this daemon. Lets look at the following examples:

Eudora Qpopper
Does the daemon access files in one general location?
Primarily, a mail popper needs to access the users' mail folders. There are some other files such as some /etc files required to perform authentication. The answer to this question is 'YES' as a majority of the files needed are in one location (/var/mail).

Is it viable to copy the files either once, or periodically using cron?
Copying mail folders is never a good idea, and copying them only once will end up giving our users stale mailboxes. Thus, it's not a good idea to copy these files at all. It is prudent, however, to copy the /etc files used and any other files used once per day or more/less frequently as needed. Since we are trying to protect these files, we don't want to give a potential hacker access to the originals.

Is it prudent to create a jail "around" a specific set of files?
There are two alternatives when dealing with files in a jail. Either move or copy the files into the jail, or build your jail around the files. In the case of mail, it is extremely resource intensive to copy mail folders back and forth every few minutes, and it is just as inefficient to try and maintain a directory full of symbolic links. It is far more efficient to get the popper to use the actual mail folders in their original location. Our answer here is 'YES'.

Sun's RPCBIND
Does the daemon access files in one general location?
The daemon primarily accesses several small /etc files.

Is it viable to copy the files either once, or periodically using cron?
Most of the files accessed are small and scarcely updated. It would be very viable to copy these files to a different directory once a day.

Is it prudent to create a jail "around" a specific set of files?
Unlike our mail popper, which accessed primarily mail, /etc contains many of the critical system files we're trying to protect. It would be very BAD and defeat the purpose of chrooting in the first place to give a potential hacker access to any files in /etc. Thus, it would be a good idea to chroot THIS daemon by having a totally separate jail, and simply copy the files over.

Sendmail
Does the daemon access files in one general location?
No. Sendmail must have access to several files in several different locations. This includes users' home directories and other files. Simultaneously, sendmail must also be able to access the mail spool (/var/spool/mqueue) and user mail directories (/var/mail).

Is it viable to copy the files either once, or periodically using cron?
In order to create a chrooted environment containing all the files necessary for sendmail to function, user directories would need to be mirrored in a secure location to contain the files necessary to redirect and manupulate mail. This would also require the creation of new directories and the deletion of old ones when user accounts and manipulated. Simultaneously, sendmail must also be able to access the mail spool and user mail folders. It is not viable to copy any of these in and out.

Is it prudent to create a jail "around" a specific set of files?
Lets take a look at this. We could create a jail around the mail folders, but then we'd lose the ability to write to the mail spool. We could jail /var, allowing access to both but then we would be allowing the chrooted environment access to potentially critical files. Neither solutions address the other issue of having to create a mirror for user directories. There doesn't seem to be any one set of files we could create a jail around.

Sendmail, as the answers suggest, is not a good candidate for chrooting. Although it may be possible, the setup would be elaborate and resource-intensive. It's a good idea to just stick to the security that comes with sendmail.

2.2 Introduction to truss, lsof, and ldd

truss (or strace)
If you 'man truss' you'll see that truss is a tool designed to trace system calls and signals. strace is a similar tool used in Linux. System calls include calls to libraries and accessing files. Truss is a great way to find out what resources a particular daemon is using. To use truss, you simply type 'truss' followed by the command you would normally use to start the daemon. Output you are looking for will look like this:

open("/usr/lib/libc.so.1", O_RDONLY) = 3
Program opened the libc library

open64("/usr/local/etc/my.config", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
Program opened /usr/local/etc/my.config for writing

Running truss on a daemon should immediately give you a long list of files to start with on your search. It's a good idea to utilize all the different features and commands of a daemon and watch for new files in truss. Without truss, you'll end up breaking your chrooted jail once for every time you're missing a data file.

lsof
lsof is another great tool for finding the resources a daemon uses. lsof is designed to operate on daemons that are already running. It will show you open files, network connections, and much more. lsof does not come with most systems, as it is a third party tool. It can be downloaded from many sites such as www.sunfreeware.com. Sources are readily available via anonymous ftp from ftp://vic.cc.purdue.edu/pub/tools/unix/lsof.

ldd
ldd is a tool handy for determing what library resources your program will need in order to execute. ldd stands for 'list dynamic dependencies'. Running ldd followed by the path and filename of the daemon you are chrooting will give you a list of libraries you must plan to copy over to the jail.

A note on dynamic libraries: If you have access to the sources for the program you are chrooting, you may wish to compile it with static libraries. This will prevent you from having to copy over libraries. If you use an operating system whose manufacturer updates the libraries frequently, however, you may wish to remain dynamic.

2.3 Determining datafile dependencies
The first little exercise we went through in 2.1 got us thinking about what datafiles the daemon is dependent on. Some are relocatable, while others we have to build a jail around. Only two of the three examples from the previous section were viable (Eudora Qpopper and Sun's RPCBIND). Both are very good candidates for chrooting and both have had severe security vulnerabilities in the past. Now lets move on to mapping out the actual datafiles that we're going to need to move or build around for these two examples. It's perfectly alright to miss a few small files as we can always find a solution for them before we turn the new service up. There are several ways to determine datafiles used. These include, but are not limited to:

The obvious (duh, a mail popper accesses mail folders)
Examining configuration files
Using tools such as 'lsof' and 'truss'
(Preferably not) Breaking the daemon later on in its jail

It's important to account for as many important datafiles as you can. Looking at our two examples, we're able to create a pretty good list of the files we're going to need:

Eudora Qpopper:
/var/mail/* (Inside jail)

The following files from /etc (copied in periodically)

hosts.allow
hosts.deny
netconfig
nsswitch.conf
passwd
resolv.conf
services
shadow

Sun RPCBIND
The following files from /etc (copied in periodically)

hosts
netconfig
nsswitch.conf
resolv.conf
services

Timezone from /usr/local/zoneinfo

A tmp directory

2.4 Creating a datafile strategy
Once you have a good list of files that will be used, keep track of how you plan on getting them into your jail. Will you copy them in? How often? Once or daily? Is your jail going to be built around them? Building a strategy will tell you which files are going to need to be copied, which ones you will build the jail around, and will start to create an idea in your head of where you want this jail to go. It is imperative that you do not build your jail around any system critical files, such as /etc, as this will alleviate the purpose of a jail. If you must build a jail around something, build it around a directory that would not necessarily cause a system failure if hacked.

If you draw a circile on paper, you can then draw all the files outside of the jail outside of the circle. Draw lines into the circle and annotate how they will be moved or copied, including how often. Any files you plan on building your jail around should be drawn inside the circle. This will start to show you how big your jail is going to need to be (filesystem path-wise) and give you some good notes to work with when it's time to create the jail.

2.5 Determining library dependencies
All the program code that your daemon uses to execute is not necessarily included in just the program file. Many peices of shared code are re-used in the form of libraries. These libraries are required by the program in order to execute properly. In a chrooted environment, the program will not have direct access to these libraries in their original locations. They will need to be initially copied into the jail in order for the program to function. The best way to determine what libraries your daemon will use is by using the command 'ldd'. ldd will list the dynamic dependencies (commonly shared libraries) required in order to execute. Take note of these, and make plans to copy them into your new jail.

Document the libraries used just as you did with your datafiles. You already know how they're going to be copied into the jail (using cp, and only once). It may be a good idea to keep this information handy in the event that you upgrade or patch your operating system, you may want to copy the updated libraries to their jail locations. An alternative would be to create a cron job to copy them over once a month.

In our examples, we've extracted the following libraries using truss:

Eudora Qpopper

/usr/lib/libnsl.so.1
/usr/lib/libsocket.so.1
/usr/lib/libresolv.so.2
/usr/lib/libmail.so.1
/usr/lib/librt.so.1
/usr/lib/libcrypt_i.so.1
/usr/lib/libc.so.1
/usr/lib/libdl.so.1
/usr/lib/libmp.so.2
/usr/lib/libaio.so.1
/usr/lib/libgen.so.1
/usr/lib/libpthread.so.1
/usr/lib/libthread.so.1

Sun RPCBIND

/usr/lib/libsocket.so.1
/usr/lib/libnsl.so.1
/usr/lib/libdl.so.1
/usr/lib/libc.so.1
/usr/lib/libmp.so.2

2.6 Finding a good home for your jail

You now have all the information necessary to determine where to set up your jail. If your jail is going to be 'built around' existing files (such as our Eudora Qpopper example), you will need to create it offset to that directory. If you do not need to create it around any files, choose an arbitrary location with significant disk space. We are going to use /var/mail for our popper and /var/rpcbind for Sun RPCBIND. Remember, your programs are not going to be know they're jailed so they're going to try and access the files just as they would normally which means they better either be where the daemon expects them to be (relatively) or symlinked. You're now ready to move onto Part III!

Part III

3.1 Creating an empty jail
The first step to creating your empty jail is to create a set of directories to hold your daemon, data files, and libraries. Lets first attack our Sun RPCBIND example. We've chosen /var/rpcbind as our jail home. Here's how we'll set up our directory structure:

mkdir /var/rpcbind
mkdir /var/rpcbind/dev
mkdir /var/rpcbind/etc
mkdir /var/rpcbind/tmp
mkdir -p /var/rpcbind/usr/lib

Since this daemon will be chrooted, it wil need access to certain /dev devices, some datafiles in etc, its own tmp directory, and a library directory. These will all be filled with appropriate files by the time we're done.

Now onto our Eudora Qpopper example. Since qpopper's jail is going to be built around our mailspool, the jail will need to start in /var/mail. We need to go ahead and create our initial directories:

mkdir /var/mail/dev
mkdir /var/mail/etc
mkdir -p /var/mail/usr/lib 
mkdir -p /var/mail/usr/local/lib
mkdir -p /var/mail/usr/sbin
mkdir -p /var/mail/usr/share/lib/zoneinfo/US

But we're not done yet. Our popper is expecting the mail to be in /var/mail. When it's chrooted, the path / is /var/mail...so when it tries to access /var/mail it will get a file not found! How do we circumvent this problem? We need to symlink /var/mail/var/mail to /var/mail which is, relatively speaking, / ...

mkdir /var/mail/var
ln -s / /var/mail/var/mail

Now when our chrooted popper goes to access /var/mail, it'll be pointed to /, which is the real /var/mail!

Our final products should look like:
Sun RPCBIND


drwxr-xr-x   7 root     other        512 Aug  1 18:31 ./
drwxr-xr-x  34 root     sys          512 Aug  1 18:07 ../
drwxr-xr-x   2 root     other        512 Aug  1 18:16 dev/
drwxr-xr-x   2 root     other        512 Aug  1 18:10 etc/
drwxr-xr-x   2 root     other        512 Aug  1 18:31 tmp/
drwxr-xr-x   6 root     other        512 Jul 28 15:55 usr/
drwxr-xr-x   3 root     other        512 Aug  1 18:19 var/

Eudora Qpopper


drwxrwxrwt   7 root     mail         512 Oct 18 19:36 ./
drwxr-xr-x  34 root     sys          512 Aug  1 18:07 ../
drwxr-xr-x   2 root     other        512 Jul 29 13:33 dev/
drwxr-xr-x   2 root     other        512 Jul 28 16:18 etc/

            - Some mailboxes mixed in here -

drwxr-xr-x   6 root     other        512 Jul 28 15:55 usr/
drwxr-xr-x   2 root     other        512 Aug  1 18:06 var/

3.2 Copying program and data files, configuring cron
Now that our directory structure is set up, lets copy over our program and data files. Since we're going to be running rpcbind from an init script, it's not necessary to copy the actual binary over for that. We will, hoever, need to copy the popper over as it will be run from inetd. If inetd expects the file to be in /usr/sbin, you will need to create a usr/sbin in your jail (/var/mail/usr/sbin) and copy the daemon's executable to there. Copy all the /etc files we previously discussed over and make sure the permissions are identical. For files you will want to update frequently, create cron jobs to update them. When you're finished, your structure should look similar to this:

Sun RPCBIND


% ls -l /var/rpcbind/etc
-r--r--r--   1 root     other         90 Jul 28 16:18 hosts
-rw-r--r--   1 root     other       1239 Jul 28 16:09 netconfig
-rw-r--r--   1 root     other        835 Jul 28 15:32 nsswitch.conf
-rw-r--r--   1 root     other        140 Jul 28 16:14 resolv.conf
-r--r--r--   1 root     other       3649 Jul 28 16:14 services

Eudora Qpopper


% ls -l /var/mail/etc

-r--r--r--   1 root     other         90 Jul 28 16:18 hosts
-rw-------   1 root     other         73 Oct 18 19:15 hosts.allow
-rw-------   1 root     other          9 Jul 28 15:58 hosts.deny
-rw-r--r--   1 root     other       1239 Jul 28 16:09 netconfig
-rw-r--r--   1 root     other        835 Jul 28 15:32 nsswitch.conf
-r--r--r--   1 root     other        815 Oct 18 19:15 passwd
-rw-r--r--   1 root     other        140 Jul 28 16:14 resolv.conf
-r--r--r--   1 root     other       3649 Jul 28 16:14 services
-r--------   1 root     other        502 Oct 18 19:15 shadow

The cron jobs we came up with look like this. Whatever works for you...

0,30 * * * * cp -p /etc/passwd /var/mail/etc/passwd
0,30 * * * * cp -p /etc/shadow /var/mail/etc/shadow
0 0 * * * cp -p /etc/hosts.* /var/mail/etc/
0 0 0 * * cp -p /etc/services /var/rpcbind/etc
0 0 0 * * cp -p /etc/resolv.conf /var/mail/etc

3.3 Copying libraries
Remember those libraries we discussed? We need to copy them to their relative places in the jail as well. Most libraries can be found in /usr/lib although some exist in /usr/local/lib. In the same fashion as your datafiles, copy the libraries over with identical permissions. You should end up with something similar to this (depending on your operating system):

% ls -l /var/rpcbind/usr/lib

-rwxr-xr-x   1 root     other     200292 Jul 28 15:27 ld.so.1*
-rwxr-xr-x   1 root     other      41628 Jul 28 15:28 libaio.so.1*
-rwxr-xr-x   1 root     other     938940 Jul 28 15:28 libc.so.1*
-rwxr-xr-x   1 root     other      15616 Jul 28 15:27 libcrypt_i.so.1*
-rwxr-xr-x   1 root     other       4448 Jul 28 15:28 libdl.so.1*
-rwxr-xr-x   1 root     other      40540 Jul 28 15:28 libgen.so.1*
-rwxr-xr-x   1 root     other      29548 Jul 28 15:27 libmail.so.1*
-rwxr-xr-x   1 root     other      19584 Jul 28 15:28 libmp.so.2*
-rwxr-xr-x   1 root     other     730672 Jul 28 15:27 libnsl.so.1*
-rwxr-xr-x   1 root     other      35308 Jul 28 15:57 libpthread.so.1*
-rwxr-xr-x   1 root     other     326336 Jul 28 15:27 libresolv.so.2*
-rwxr-xr-x   1 root     other      39048 Jul 28 15:27 librt.so.1*
-rwxr-xr-x   1 root     other      65876 Jul 28 15:27 libsocket.so.1*
-rwxr-xr-x   1 root     other     166624 Jul 28 15:58 libthread.so.1*
-rwxr-xr-x   1 root     other      19648 Jul 28 16:17 nss_dns.so.1*
-rwxr-xr-x   1 root     other      38832 Jul 28 15:33 nss_files.so.1*
-rwxr-xr-x   1 root     other      38292 Jul 28 15:33 nss_nis.so.1*
-rwxr-xr-x   1 root     other      12284 Aug  1 18:15 straddr.so*

% ls -l /var/mail/usr/lib /var/mail/usr/local/lib

-rwxr-xr-x   1 root     other     200292 Jul 28 15:27 ld.so.1*
-rwxr-xr-x   1 root     other      41628 Jul 28 15:28 libaio.so.1*
-rwxr-xr-x   1 root     other     938940 Jul 28 15:28 libc.so.1*
-rwxr-xr-x   1 root     other      15616 Jul 28 15:27 libcrypt_i.so.1*
-rwxr-xr-x   1 root     other       4448 Jul 28 15:28 libdl.so.1*
-rwxr-xr-x   1 root     other      40540 Jul 28 15:28 libgen.so.1*
-rwxr-xr-x   1 root     other      29548 Jul 28 15:27 libmail.so.1*
-rwxr-xr-x   1 root     other      19584 Jul 28 15:28 libmp.so.2*
-rwxr-xr-x   1 root     other     730672 Jul 28 15:27 libnsl.so.1*
-rwxr-xr-x   1 root     other      35308 Jul 28 15:57 libpthread.so.1*
-rwxr-xr-x   1 root     other     326336 Jul 28 15:27 libresolv.so.2*
-rwxr-xr-x   1 root     other      39048 Jul 28 15:27 librt.so.1*
-rwxr-xr-x   1 root     other      65876 Jul 28 15:27 libsocket.so.1*
-rwxr-xr-x   1 root     other     166624 Jul 28 15:58 libthread.so.1*
-rwxr-xr-x   1 root     other      19648 Jul 28 16:17 nss_dns.so.1*
-rwxr-xr-x   1 root     other      38832 Jul 28 15:33 nss_files.so.1*
-rwxr-xr-x   1 root     other      38292 Jul 28 15:33 nss_nis.so.1*

3.4 Creating Devices
Here comes the tricky part. With the daemon chrooted, it will not be able to access the traditional /dev directory, which includes several commonly used devices such as /dev/null, /dev/log, and etcetera. Finding out what devices your daemon uses is no walk in the park either. You can find most out by using truss and sometimes strings | grep dev, but for the most part you'll have to watch for error messages if you are missing a device. There are a general set of devices that should be created for any daemon:


/dev/conslog
/dev/log
/dev/msglog
/dev/null
/dev/tcp
/dev/ticlts
/dev/ticots
/dev/ticotsord
/dev/udp
/dev/zero

Here's how to create your devices (do this for each device):
1. do an 'ls -lL /dev/[devicename]'. The output will look like this:
crw-rw-rw- 1 root sys 13, 2 Oct 18 19:56 /dev/null

The number in red is your 'major' number. The number in blue is your 'minor' number. The first letter, which is in green will be either a 'c' or a 'b' signifying whether the device is a character or block device. With all this information, you can now create the node.

2. cd into the jail's dev directory (e.g. /var/mail/dev or /var/rpcbind/dev). Use the command 'mknod' to create the device. For example: mknod null c 13 2 would create the device we listed in our example. Most machines have different major/minor numbers so be careful.

3. using chown and chmod, make sure the device reflects identical permissions as the original device.
For our Sun RPCBIND example, we ended up breaking it later on to discover that the creation of a directory called rpc_door with the sticky bit set needed to be created. This may not be the case on your operating system, but if you come across something like that you'll need to create it:

mkdir -p /var/rpcbind/var/run/rpc_door
chmod +t /var/rpcbind/var/run/rpc_door

Blah.

3.5 Altering startup scripts
Now you should be done! If you missed something, you'll likely hear about it when you first run the daemon to test it. You did test it didn't you? Oh wait, we forgot to tell you how! Well, before you can safely test most daemons, you need to turn their non-chrooted version off. Once you've killed the production daemon, you can execute the command like this:

/usr/sbin/chroot /var/rpcbind /usr/sbin/rpcbind
Hopefully everything should run fine. If you get complaints of missing files or libraries, you can use truss to determine the filename, and copy it into your path. Once you're up and running, you can simply comment out the old startup command in your startup scripts and replace it with a chrooted version.

Chrooting from inetd
Eudora Qpopper runs from inetd, and therefore needs its own commandline to start it up chrooted every time. To do this, use a line similar to:

pop3 stream tcp nowait root /usr/sbin/chroot chroot /var/mail /usr/sbin/in.pop3

Or if you're using TCP wrappers, you'll need to "trick" inetd into thinking its service name is still in.pop3 (or whatever you use). In order to do this, symlink your service name to chroot and then use the service name in inetd.conf. For example:

lrwxrwxrwx 1 root other 6 Jul 28 15:42 /usr/sbin/in.pop3 -> chroot*

pop3 stream tcp nowait root /usr/sbin/tcpd in.pop3 /var/mail /usr/sbin/in.pop3

/usr/sbin/in.pop3 is *actually* /usr/sbin/chroot (symlinked), so when inetd starts the service in.pop3, chroot actually runs with the parameters '/var/mail /usr/sbin/in.pop3' which will run the chrooted version of in.pop3 located in /var/mail. When the chrooted copy is started up by chroot, the service name will be set correctly.

3.6 The Final product

Our final, working projects looks like this:
Sun RPCBIND


d none /var/rpcbind 0755 root other
d none /var/rpcbind/dev 0755 root other
c none /var/rpcbind/dev/conslog 21 0 0666 root other
c none /var/rpcbind/dev/log 21 5 0640 root other
c none /var/rpcbind/dev/msglog 97 1 0600 root other
c none /var/rpcbind/dev/null 13 2 0666 root other
c none /var/rpcbind/dev/udp 41 0 0666 root other
c none /var/rpcbind/dev/tcp 42 0 0666 root other
c none /var/rpcbind/dev/ticlts 105 2 0666 root other
c none /var/rpcbind/dev/ticotsord 105 1 0666 root other
c none /var/rpcbind/dev/ticots 105 0 0666 root other
d none /var/rpcbind/var 0755 root other
d none /var/rpcbind/var/run 0755 root other
d none /var/rpcbind/var/run/rpc_door 1777 root root
d none /var/rpcbind/tmp 0755 root other
d none /var/rpcbind/usr 0755 root other
d none /var/rpcbind/usr/share 0755 root other
d none /var/rpcbind/usr/share/lib 0755 root other
d none /var/rpcbind/usr/share/lib/zoneinfo 0755 root other
d none /var/rpcbind/usr/share/lib/zoneinfo/US 0755 root other
f none /var/rpcbind/usr/share/lib/zoneinfo/US/Eastern 0644 root bin
d none /var/rpcbind/usr/lib 0755 root other
f none /var/rpcbind/usr/lib/ld.so.1 0755 root other
f none /var/rpcbind/usr/lib/libnsl.so.1 0755 root other
f none /var/rpcbind/usr/lib/libsocket.so.1 0755 root other
f none /var/rpcbind/usr/lib/libresolv.so.2 0755 root other
f none /var/rpcbind/usr/lib/libmail.so.1 0755 root other
f none /var/rpcbind/usr/lib/librt.so.1 0755 root other
f none /var/rpcbind/usr/lib/libcrypt_i.so.1 0755 root other
f none /var/rpcbind/usr/lib/libc.so.1 0755 root other
f none /var/rpcbind/usr/lib/libdl.so.1 0755 root other
f none /var/rpcbind/usr/lib/libmp.so.2 0755 root other
f none /var/rpcbind/usr/lib/libaio.so.1 0755 root other
f none /var/rpcbind/usr/lib/libgen.so.1 0755 root other
f none /var/rpcbind/usr/lib/nss_files.so.1 0755 root other
f none /var/rpcbind/usr/lib/nss_nis.so.1 0755 root other
f none /var/rpcbind/usr/lib/libpthread.so.1 0755 root other
f none /var/rpcbind/usr/lib/libthread.so.1 0755 root other
f none /var/rpcbind/usr/lib/nss_dns.so.1 0755 root other
f none /var/rpcbind/usr/lib/straddr.so 0755 root other
d none /var/rpcbind/usr/sbin 0755 root other
f none /var/rpcbind/usr/sbin/rpcbind 0555 root other
d none /var/rpcbind/usr/local 0755 root other
d none /var/rpcbind/usr/local/sbin 0755 root other
d none /var/rpcbind/usr/local/lib 0755 root other
d none /var/rpcbind/etc 0755 root other
f none /var/rpcbind/etc/nsswitch.conf 0644 root other
f none /var/rpcbind/etc/netconfig 0644 root other
f none /var/rpcbind/etc/services 0444 root other
f none /var/rpcbind/etc/resolv.conf 0644 root other
f none /var/rpcbind/etc/hosts 0444 root other

Eudora Qpopper


d none /var/mail 1777 root mail
d none /var/mail/usr 0755 root other
d none /var/mail/usr/share 0755 root other
d none /var/mail/usr/share/lib 0755 root other
d none /var/mail/usr/share/lib/zoneinfo 0755 root other
d none /var/mail/usr/share/lib/zoneinfo/US 0755 root other
f none /var/mail/usr/share/lib/zoneinfo/US/Eastern 0644 root bin
d none /var/mail/usr/lib 0755 root other
f none /var/mail/usr/lib/ld.so.1 0755 root other
f none /var/mail/usr/lib/libnsl.so.1 0755 root other
f none /var/mail/usr/lib/libsocket.so.1 0755 root other
f none /var/mail/usr/lib/libresolv.so.2 0755 root other
f none /var/mail/usr/lib/libmail.so.1 0755 root other
f none /var/mail/usr/lib/librt.so.1 0755 root other
f none /var/mail/usr/lib/libcrypt_i.so.1 0755 root other
f none /var/mail/usr/lib/libc.so.1 0755 root other
f none /var/mail/usr/lib/libdl.so.1 0755 root other
f none /var/mail/usr/lib/libmp.so.2 0755 root other
f none /var/mail/usr/lib/libaio.so.1 0755 root other
f none /var/mail/usr/lib/libgen.so.1 0755 root other
f none /var/mail/usr/lib/nss_files.so.1 0755 root other
f none /var/mail/usr/lib/nss_nis.so.1 0755 root other
f none /var/mail/usr/lib/libpthread.so.1 0755 root other
f none /var/mail/usr/lib/libthread.so.1 0755 root other
f none /var/mail/usr/lib/nss_dns.so.1 0755 root other
d none /var/mail/usr/sbin 0755 root other
f none /var/mail/usr/sbin/in.pop3 0755 root other
d none /var/mail/usr/local/lib 0755 root other
d none /var/mail/etc 0755 root other
f none /var/mail/etc/passwd 0444 root other
f none /var/mail/etc/shadow 0400 root other
f none /var/mail/etc/hosts.allow 0600 root other
f none /var/mail/etc/nsswitch.conf 0644 root other
f none /var/mail/etc/hosts.deny 0600 root other
f none /var/mail/etc/netconfig 0644 root other
f none /var/mail/etc/services 0444 root other
f none /var/mail/etc/resolv.conf 0644 root other
f none /var/mail/etc/hosts 0444 root other
d none /var/mail/var 0755 root other
s none /var/mail/var/mail=/
d none /var/mail/dev 0755 root other
c none /var/mail/dev/udp 41 0 0666 root other
c none /var/mail/dev/null 13 2 0666 root other
c none /var/mail/dev/conslog 21 0 0666 root other
c none /var/mail/dev/log 21 5 0640 root other
c none /var/mail/dev/msglog 97 1 0600 root other

Daemons I've personally chrooted without problems:

popper
rpcbind
named (It's pretty much built-in)
stunnel
Apache web server
nscd
SecurID Server
RADIUS
Squid
syslogd

Daemons that theoretically should be chrootable, though I haven't tried:

portmap
lpd (with slight creativity)
gated
routed
identd
comsat
talkd
tftpd
sadmind
Many other processes that should probably just be turned off

3.7 Logging with syslog

The syslog facility on most System V based systems uses /dev/log and possibly other /dev/*log devices. The nice thing about this is that you can still effectively syslog by re-creating these devices in your chrooted environment. I've tested this on Solaris and it worked. Non-SysV systems such as linux may not have this functionality and will therefore need to send syslog packets to itself (machine).

Part IV: Advanced Topics

4.1 How to tell if you're chrooted

Some have asked how one can tell if you are in a chrooted environment. Every mountpoint in a ufs filesystem has an inode number of 2. This means if your jailed / is not on a mountpoint, the inode number will be different. Since many systems administrators implement chroot() without considering this, the following code could tell you if you are chrooted and not on a mountpoint.

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>

int main(int argc, char **argv) {
  struct stat x;

  if (stat("/", &x)) {
    printf("Unable to stat /");
    exit(EXIT_FAILURE);
  }

  if (x.st_ino==2) {
    printf("I am not chrooted or chrooted on a mountpoint\n");
  } else {
    printf("I am chrooted\n");
  }
  exit(EXIT_SUCCESS);
}

Obviously, if you lack a complete system environment (e.g. no shadow password file, 'su' binary, or other obvious giveaways), it is easy to tell if you are in a chroot()ed environment. Some administrators configure separate full system environments for their users, however, in which they don't want their users to know they're chroot()ed. If the / inode is 2 and you still suspect you are in a chroot()ed environment, you can check the following:

Check for files such as a shadow password file and 'su' binary.
Perform a process list and check for the existence of the process' binaries (some OS's allow the administrator to hide processes as well)
If your OS has a /proc, check for a /proc
Perform a 'df' and check to see if any other filesystems are mounted; if so, can you see them? if not, very suspicious.

4.2 Breaking out of a chroot()ed cell

Breaking out of a properly chrooted cell can be made quite difficult, but with the right tools is possible from within your cell. In order for someone to be able to break out of your chrooted cell, they must be able to:

Find a vulnerability in the software you're running in the cell
Exploit this vulnerability and find a way to upload arbitrary code into memory or disk
If uploaded into memory, find a way to execute it as root
If uploaded to disk, also upload a shell into your cell and call the shell to execute the code as root

Uploading and executing the arbitrary code into memory can be as simple as uploading and executing root shell code as done in many exploits. The complete code to break out of a chroot()ed cell is only a mere 139 bytes! Once they've accomplished this, breaking out of the cell is relatively simple. Notice, however, that the above scenario is only possible if the software you are running is running as root. It is strongly recommended, to make chroot more effective, that you run your system software as a non-privileged user. This makes it much more difficult to break out of a cell.

Breaking out is beyond the scope of this document, but the following link is a pretty good explanation of one method of doing it: http://www.bpfh.net/computing/docs/chroot-break.html With careful planning and understanding of chroot()'s vulnerabilities, it should be possible to make breaking out extremely difficult. There are also some great tools available for intercepting system calls (such as root shell calls) to harden your OS.

4.3 Frequent chroot() Mistakes

Even in a chroot()ed environment, running the target software with certain loopholes can seriously degrade the security that chroot() provides. The following is a growing list of common mistakes many administrators can make when chroot()ing processes:

Mistake: Running chroot()ed processes with root privileges
Severity: Critical

One of the most common misperceptions is that running your software as root in a chroot()ed environment is safe. This is completely untrue. The idea behind chroot() is to protect your system from someone exploiting a vulnerability in your target software. If your software indeed does have a vulnerability, running the software as root provides many ways for your system to be compromised. Sure, chroot() may protect you from the average script kiddie, but an experienced hacker can easily break out of the cell (if they have root privileges), play with your chroot()ed /dev devices, kill, trace, or snoop on other processes, and wreak general havoc all from within their cell if the target software is running as root.

Solution: Modify your implementation to change its user and group id's after chroot()ing. This can be done either by manually modifying the source code to perform setuid() and setgid() function calls, or by writing a small wrapper to do it. Since this software is running in a chroot()ed environment, you should be able to safely change file permissions around to accomodate a non-root user. For example, if you are chroot()ing popper, you can modify the popper to run as nobody, but in the group owning all mailboxes. This will enable you to service mail requests without being super user. I have written a small patch to accomplish this.

Mistake: Using a copy of the shadowed password file
Severity: High

In our popper example, we copied the shadow password file into the chroot()ed directory to provide authentication. Unfortunately, if there is a vulnerability in the popper, it may allow a hacker to steal the shadow password file, run a cracker, and gain access to the system through the front door.

Solution: There are a few different solutions to this. First, if you are able to implement an external method of authentication (for example RADIUS, SecurID, or even utilizing the PAM libraries) that will allow you to authenticate without having a copy of the shadow file inside the jail, this is the best solution. If this is impractical for your implementation, another way to handle this would be to parse the shadow password file prior to copy and remove any system or privileged (shell) accounts from the file. If some privileged accounts must be used, set up your shadow file to use different passwords for email than used for the system.

Mistake: Creating virtual system environments off of a mountpoint
Severity: Low

By chroot()ing a directory that is not also a mountpoint, you provide a small hole for detecting that the user is chroot()ed. As we discussed earlier, all mountpoints have an inode number of 2. By chroot()ing off of a mountpoint, you advertise that the process is chroot()ed. This is not very dangerous as it doesn't provide a way to break out of the chroot() but is nevertheless providing some information the hacker shouldn't have. If you have a relatively small chroot()ed environment where the lack of several system files makes it obvious that you are chroot()ed, there is no reason to correct this, but if you are trying to emulate a complete environment (e.g. virtual machines), it may be a good idea to put your jail on a mountpoint.

Solution: If you are trying to emulate a full system environment (e.g. virtual machines) and do not want your users to know they're chroot()ed, place the jail on a mountpoint.

Reproduction prohibited without permission