- Any suggestions on where I can lay my hands on the ALPINE src code and their documentation. The project is conspicuously missing on the web.
As far as I can tell Alpine and Alpine4Linux were user-space ports of the FreeBSD 4.8 TCP networking stack
The URL's were
http://alpine.cs.washington.edu/ http://www.vzavenue.net/~neelnatu/alpine4linux/. Unfortunately, these doesn't exist any more.
I found a paper on it:
http://www.usenix.org/event/usits01/full_papers/ely/alpineUs.pdf
If anyone has the code, please let me know, send me a copy if possible.
Meanwhile if I ever get my dead server I will try to recover my copy off the disks.
I would really love to have this code.
List: linux-net Subject: Re: Is it possible to run TCP/IP stack in user space? From: John HeffnerDate: 2006-11-09 5:29:14 Subramonia Pillai wrote: > Hi, > > I have one doubt. Is it possible to run a third party > TCP/IP stack as application over a native linux > kernel. Please give some pointer how it will be done? Maybe, with some difficulty. I'm aware of a few things that did this over BSD, but they're really out of date. The Alpine project pulled the BSD TCP stack out into a userspace library: <http://sosp16.cs.washington.edu/homes/djw/papers/Usits01.pdf>. The FoxNet project wrote a full TCP/IP stack from the ground up in SML: <http://www.cs.cmu.edu/~fox/foxnet.html>. IIRC, they both had similar hack layers for running on top of a unix kernel, using pcap, firewall filters and raw sockets.
Harvested from Archive.org, looks like appeared in 2003 then vanished in 2005.
Alpine4Linux
Neelkanth Natu (neelnatu@yahoo.com)
Alpine4Linux consists of a userland server program that runs the FreeBSD kernel code as well as the unmodified networking stack. Client programs use the Alpine stack by setting the LD_PRELOAD environment variable to link to libraries, that intercept socket related system calls. These intercepted system calls are routed to the Alpine server over a TCP connection established on the loopback interface.
Alpine4Linux goes great lengths to ensure that its behavior is identical to that of a FreeBSD kernel as far as networking is concerned. Thus, in addition to the unmodified FreeBSD stack. Alpine4Linux also has unmodified socket layer code, file descriptor code, tsleep and wakeup for e.g. It also has a rich client-side library that supports almost all socket functions as well as functions like fork() that are commonly used by server programs.
Alpine4Linux Server in Action
Here is a output of the Alpine4Linux server when it starts up. The server initializes the kernel when it starts up which is the reason for the FreeBSD copyright messages.[root@localhost scripts]# bash run_alpine_server.sh
Modifying INPUT and FORWARD chains to drop packets destined to 10.11.12.13
Listening on 127.0.0.1:8475!
Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 4.8-RELEASE #4: Fri May 30 02:29:53 PDT 2003
neel@localhost:/home/neel/av/main/bsd/src/sys/compile/NINT
Opening tap0
Setting hwaddr for tap0 device to: 0:4:76:ec:2a:f0
Setting IP address of tap0 device to: 10.11.12.13
Alpine4Linux uses the "tap0" pseudo-device within the FreeBSD kernel to do raw packet I/O. The "tap0" interface is mapped to a real Linux device (e.g. "eth0") on which the packet I/O really happens. The server also sets the hardware address of the "tap0" pseudo-device to match the hardware address of the real device "eth0". This allows the FreeBSD stack to reply to arp requests for its IP address with a valid hardware address. Note that we change iptables ruleset so that the Linux kernel will ignore packets destined to the Alpine stack.
Alpine4Linux Clients in Action
We can run networking utilities like "ifconfig" and "route" against the Alpine4Linux stack. Note that the "ifconfig" program is the stock Linux program. Also note that we are using a wrapper shell script to execute the programs. The shell script sets up the LD_PRELOAD and LD_LIBRARY_PATH variables before launching the program.[neel@localhost scripts]$ bash run_alpine_client.sh /sbin/ifconfig tap0
tap0 Link encap:Ethernet HWaddr 00:04:76:EC:2A:F0
inet addr:10.11.12.13 Bcast:10.11.12.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
[neel@localhost scripts]$ bash run_alpine_client.sh ../sbin/route/route -- add 10.11.12.0/24 -interface 10.11.12.13
add net 10.11.12.0: gateway 10.11.12.13
[neel@localhost scripts]$ bash run_alpine_client.sh ../sbin/route/route -- get 10.11.12.1
route to: 10.11.12.1
destination: 10.11.12.0
mask: 255.255.255.0
interface: tap0
flags:
recvpipe sendpipe ssthresh rtt,msec rttvar hopcount mtu expire
0 0 0 0 0 0 1500 -12
I have also tested client programs like nmap, telnet and server programs like vsftpd-1.1.3 successfully against Alpine4Linux.
Source Code
You can download the source code here. Untar the tarball using 'tar xvzf alpine4linux.tar.gz' and look for the file docs/running.txt to start playing with Alpine4Linux.Documentation
Get Alpine4Linux up and running quick: running.txtA description of how Alpine4Linux pretends to be the FreeBSD kernel: alpine4linux.txt
FAQ: faq.txt
Limitations of Alpine4Linux: limitations.txt
Differences between Alpine4Linux and the original Alpine implementation: differences_from_alpine4bsd.txt
Acknowledgements
David Ely, Stefan Savage, David Wetherall for AlpinePerforce source control system for their free 2-client license
Shri and Ravi for tolerating me when I am babbling
Introduction: Alpine4Linux is a userlevel FreeBSD 4.8 networking stack running on top of a stock Linux kernel. The original idea is attributed to [1]. However I must point out the Alpine4Linux is *not* a port of the original Alpine (referred to as Alpine4BSD henceforth). In fact there is not a single line of code common to the two implementations. Alpine4Linux has two components: 1. A daemon (alpine_server), that runs the FreeBSD stack code and does network I/O on behalf of processes wishing to use the FreeBSD stack. 2. Shared libraries (libClientSocket.so and libAlpineSupport.so) that hijack networking related system calls and divert them to the alpine_server. Why Alpine4Linux: I did this project because I was fascinated by the idea of running kernel components in a userlevel process. But seriously, I don't know why anyone would want to run a FreeBSD stack in userspace on a Linux box. The authors of the original Alpine4BSD paper cite better debugging and faster compile-test cycles when doing network protocol development. That seems to be as good a reason as any. Supported versions: The FreeBSD stack is the 4.8-RELEASE version (downloaded on April 8 2003). The only Linux dependency that the code has is that it support PF_PACKET family of sockets and it should support makecontext() and swapcontext(). Other than that it should run on any Linux distribution. uname -a on my Linux box: Linux localhost 2.4.20-13.8 #1 Mon May 12 12:20:54 EDT 2003 i686 i686 i386 GNU/Linux How does it work: The alpine_server is a Linux program that acts like a FreeBSD kernel as far as the networking stack is concerned. Just like the FreeBSD kernel it provides client programs with a socket layer. It also does network I/O on behalf of client programs. But that is where the similarity ends. A kernel presents a system call interface to client programs. The alpine_server presents a RPC interface. RPC here simply means that it listens for requests over the network. For e.g. if the client program tries to open a socket(), a message will be sent to the alpine_server. The message will indicate the type of the request (REQ_SOCKET) and its parameters (AF_INET,SOCK_STREAM). The response will contain the type of the response (RESP_SOCKET) and the return value (socket_fd, errno). Client programs link with 2 shared libraries libClientSocket.so and libAlpineSupport.so using LD_PRELOAD and LD_LIBRARY_PATH environment variables. These libraries "intercept" the socket related functions before they can be processed conventionally by the Linux libc. Instead each socket related system call (e.g. socket(), bind(), connect()) is transformed into a message to the alpine_server. The alpine_server and its clients communicate over a standard TCP socket bound to 127.0.0.1. Requirements: An unmodified networking stack: The sys/net, sys/netinet and certain files under sys/kern must be from the stock FreeBSD-4.8 release. This requirement was mostly satisfied. I had to make 3 changes to work around some differences between Linux and FreeBSD. The changes are trivial and do not change functionality. See Appendix A for more details on these changes. Ability to use unmodified Linux binaries: It should be possible to simply define the LD_PRELOAD and LD_LIBRARY_PATH variables and run any dynamically linked Linux networking application against the FreeBSD stack. E.g. It is possible to configure the FreeBSD stack interfaces using the stock 'ifconfig' on Linux. I have also run 'telnet', 'nmap' and 'ping' against the alpine_server with no problems. Unfortunately it was too difficult to use the Linux 'route' command against the FreeBSD stack. Alpine4Linux provides the 'route' program from FreeBSD, ported to Linux, that can be used to configure routing in the FreeBSD stack. Reuse as much of the FreeBSD kernel code as possible: This requirement is subjective but I think Alpine4Linux utilizes a *lot* of unmodified FreeBSD kernel code. In fact the alpine_server defines only two non-trivial functions that are required by the kernel: mi_switch() and scheduler(). scheduler() runs the main select() loop in the alpine_server. mi_switch() deals with switching the FreeBSD kernel execution context. These functions are described in detail later in this document. Alpine4BSD uses unmodified sysinit, timeouts, tsleep() and wakeup(), descriptor management for e.g. Implementation: Sending and receiving packets: The alpine_server is invoked with the name of the interface (on the host OS) that it uses to send/receive packets. The IP address and subnet that are used by the FreeBSD stack are also specified on the command line. E.g. ./alpine_server eth0 10.11.12.13 255.255.255.0 This tells the alpine_server to use the "eth0" interface on Linux to send/receive packets. It also assigns 10.11.12.13/24 as the IP address of the FreeBSD stack. The alpine_server first opens a socket of family PF_PACKET. This is the recommended way to do raw packet I/O on Linux. A BPF program is compiled, so that only packets destined for the FreeBSD stack are injected into the stack. The BPF filter expression is "host". Lets call this file descriptor the 'linux_pcap_fd'. Next we open the "tap" pseudo-device in the FreeBSD stack. This device presents an Ethernet device interface to the FreeBSD stack. On the other side the "tap" device returns an 'fd' that can be read and written to inject raw ethernet packets into the FreeBSD stack. Lets call this file descriptor the 'freebsd_tap_fd'. The alpine_server now configures this "tap" device by setting its MAC address to the MAC address of the interface specified on the command line. It also sets the IP address of the "tap" device to that specified on the command line. Now the job of the alpine_server is simply to read a packet from 'linux_pcap_fd'; run the packet through the BPF filter, and write the packet to 'freebsd_tap_fd'. In the other direction it reads from 'freebsd_tap_fd' and writes to 'linux_pcap_fd'. Simulating interrupts: The alpine_server puts the 'linux_pcap_fd' in its select() read fdset. Whenever a packet arrives at the interface, select() returns and the packet can be read, filtered and injected into the FreeBSD stack. Alpine4Linux acts like a true interrupt driven stack because we inject packets into the FreeBSD stack as and when we get them. Software interrupts: The alpine_server has only one thread of control running at any point in time. There is no need to lock data structures because this thread of control cannot be preempted; it has to voluntarily relinquish CPU by calling mi_switch(). Thus all the spl* functions are no-ops in Alpine4Linux. setsoftnet() is also a no-op in Alpine because we run the netisrs periodically. In Alpine4Linux this happens at every tick (1/HZ secs). The function do_netisrs() defined in kern/kern_netisr.c is called periodically by the alpine_server. This function calls all the netisrs ready to run, and gives them the opportunity to drain packets from their packet queues. Initialization: Alpine4Linux initializes the kernel data structures as if the kernel had booted itself. The alpine_server contains main() that is the entry point into the program. main() in turn calls init386() followed by mi_startup(). init386(): init386() was rewritten to only initialize the tunable variables in the kernel like "hz" or "tick". It also initializes physical memory dependent variables like "maxusers" and "maxproc". Alpine4Linux makes the FreeBSD kernel believe that it is running on a machine with 1Gbytes of physical memory. mi_startup(): This is the stock mi_startup() from the FreeBSD kernel, since we support sysinit in libAlpineSys.so. This function does not return and control ends up in the scheduler() function. Alpine4Linux defines the scheduler() function in alpine_server. Eventually control lands in the main select() loop defined in sched_main_loop(). Timer management: Timeout: Timer management in Alpine4Linux is very simple. In the main select() loop, we call hardclock() every 10 msec (this interval is based on kern.hz). If there is any event in the current timer wheel bucket, softclock() is called from hardclock(). At that point the stock FreeBSD code is used to deal with timeout events. slowtimo() and fasttimo() are indirectly called using this mechanism. tsleep() and wakeup(): Alpine4Linux uses the stock tsleep() and wakeup() functions from FreeBSD without any modifications. The blocking behavior of a process in the kernel is implemented by mi_switch() that is defined outside the kernel. Multiple execution contexts in the stack: The alpine_server provides networking services to multiple clients at the same time. It is thus imperative that the alpine_server not block in the kernel. This is the same constraint that the FreeBSD kernel itself operates under. Anytime a client process does an action that causes it to block (e.g. a blocking read() on a socket), the alpine_server must store the execution context and switch to another client process that is ready to run. If there are no client processes ready to run, the alpine_server blocks in select(). The select() loop of alpine_server is analogous to the idle loop of a Unix kernel. The alpine_server provides multiple execution contexts (one for each client), using the makecontext(3) function available in Linux. It switches between execution contexts in the FreeBSD kernel using swapcontext(3). The alpine_server itself executes in a 'ucontext_t' that is accessible as a global variable (sched_thread->ut_ctx). The alpine_server (and hence the FreeBSD stack) executes in this context for system level events like timeouts, network I/O etc. The alpine_server executes in a 'ucontext_t' associated with a client process whenever it is executing code in the FreeBSD kernel on behalf of the client process. For e.g. if the client process sends a messages to the alpine_server to read() from a socket, the alpine_server first creates a 'ucontext_t' and switches execution to the newly created context. If all goes well and there is data to be read, we will reply back to the client_process; the newly created 'ucontext_t' will be destroyed and control passes back to the main 'sched_thread' context. If there is not enough data to be read, then the ucontext_t will need to block in tsleep() and it will call mi_switch(). mi_switch() does a swapcontext() to the main 'sched_thread'. The sched_thread either idles on select() or does a swapcontext() to process a request from another client. The alpine_server also has to take care of "woken-up" execution contexts. For e.g. Consider an execution context that was put to sleep because there was not enough data to satisfy a read(). When data arrives on that socket, that execution context becomes runnable (i.e. p->p_stat == SRUN). We check for such "woken-up" processes just before the sched_thread sleeps in select(). It traverses all the ucontexts that are sleeping state *but* their proc structure is runnable i.e.(ut->ut_state==UTS_SLEEPING && p->p_stat==SRUN). if such a ucontext is found it does a swapcontext() to it. The blocked ucontext resumes execution after the mi_switch() statement in tsleep() just like it would in a stock FreeBSD kernel. Interaction with the Linux kernel: Alpine4Linux is a pure userlevel process and requires no Linux kernel modifications. But Alpine4Linux is handing out file descriptors to client programs (fd = socket()); it needs to ensure it does not step on the Linux kernel's toes when it does so. Therefore we need to map file descriptors between the FreeBSD stack and the host OS. To see why we need this, consider a case where an application issues a socket() system call. This call is intercepted by the libClientSocket library and a file descriptor is assigned to the newly created socket by the FreeBSD stack. Lets call this file descriptor the 'alpine_fd'. We need to ensure that the value we return to the client application is an fd that is not already been allocated and will not be allocated in the future. Hence we open("/dev/null") on the host OS and create a mapping between linux_fd and alpine_fd. The fd that is returned to the client from the socket() system call is the linux_fd. When the client comes back to do read() or write() with the linux_fd, we will map that fd to the alpine_fd and use it in the FreeBSD stack. Alpine4Linux compared to Alpine4BSD: Look at the file "differences_from_alpine4bsd.txt" under the docs/ directory for salient differences between Alpine4Linux and Alpine4BSD. Performance: I have not done any performance measurements for Alpine4Linux, because I am confident that its performance sucks! Since Alpine4Linux does message passing between the client program and alpine_server there are a lot of copies of when reading or writing data. Future work: Support IPv6, IPSEC etc. References: [1] Alpine: A User-Level Infrastructure for Network Protocol Development David Ely, Stefan Savage, David Wetherall http://alpine.cs.washington.edu/ Appendix A: The following are the descriptions of the 3 changes I had to make in the FreeBSD stack to make it work on Linux. All changes are trivial and do not affect functionality. netinet/if_ether.c: printf on Linux does not have the %D modifier netinet/in.c: ifconfig on Linux do not zero out sin_zero in sockaddr_in when doing SIOCSIFADDR, SIOCSIFDSTADDR and SIOCSIFBRDADDR. This causes problems when binding to that IP address, because ifa_ifwithaddr() compares the entire 16 bytes; but there is garbage in the last 8 bytes of ifa->ifa_addr. The fix was to zero out sin_zero of ia->ia_addr, ia->ia_dstaddr and ia->ia_broadaddr in in_ifinit(). net/if.c: We map the Linux SIOCSIFHWADDR to the FreeBSD SIOCSIFLLADDR. However Linux does not define the sa_len member in its sockaddr structure. We should not return EINVAL if if the 'sa_len' does not match 'sdl->sdl_alen' in if_setlladdr().
Introduction: This file describes the limitations of Alpine4Linux. Limitations: Alpine4Linux cannot pass file descriptors from one process to another. Some programs (e.g. vsftpd) use this feature; a privileged process binds a socket to a privileged port and then transfers this fd to a less privileged process to do the actual data transfer. Therefore vsftpd "active" mode does not work with Alpine4Linux.
Introduction: This file describes the differences between the original Alpine implementation and Alpine4Linux. The original Alpine is referred to as Alpine4BSD henceforth. Differences: Alpine4Linux needs a unique IP address separate from the host OS. Alpine4BSD shares the IP address already assigned to the host OS. I chose this approach because: 1. It is simpler (I did not have to write code to share the port-space with the host kernel) 2. Alpine4Linux will be used in a research environment where assigning the host OS an additional IP address should not be difficult. As long as the host OS does not send RSTs or ICMP unreach messages to the sender, we should be fine. Such a "blackhole" behavior can be configured on a Linux box using iptables. Alpine4Linux provides a helper script - "run_alpine_server.sh" - that configures iptables before starting alpine_server. It also cleans up when alpine_server exits. Alpine4Linux support fork()! This is probably the biggest difference from Alpine4BSD. Alpine4BSD uses a faux-ethernet device that was newly written to inject packets into and get packets from the FreeBSD stack. I leveraged the "tap" pseudo-device already present in the FreeBSD stack to achieve identical functionality. No code change were made to the "tap" driver. Alpine4BSD requires support for BPF devices to be compiled in the host OS. Alpine4Linux requires support for PF_PACKET sockets in the host OS. Alpine4BSD simulates interrupts by polling pcap once every 1ms by using SIGALRM. We don't have to do that since we put the 'linux_pcap_fd' in select(), so alpine_server is woken up every time a packet is available to read on the pcap_fd. In this sense Alpine4Linux is a true interrupt driven stack.
Q. I don't want the alpine_server to listen on 127.0.0.1:8475. How do I tell
it to listen for requests on another address?
A. Define the environment variables ALPINE_SERVER_LISTEN_ADDR and
ALPINE_SERVER_LISTEN_PORT appropriately. Make sure that the client programs
also execute in an enviroment with the same variables defined.
Q. I can't ping the Alpine4Linux IP address from the same machine that it
is running on.
A. Let me describe the setup first, the problem next and the solution
last.
The setup:
Consider that the native linux machine has an interface eth0 with an IP
address 10.11.12.1. We run Alpine4Linux on the same interface with an IP
address 10.11.12.13 (./alpine_server eth0 10.11.12.13 255.255.255.0).
The problem:
Now if we try to ping from the Linux box to 10.11.12.13 we won't be able
to "see" the Alpine stack. This is because the ARP request from the Linux
box is thrown away by the Alpine stack because it has a source hardware
address that Alpine considers to be its own. It believes that it is
seeing an echo of its own ARP request and the packet is discarded.
The solution:
1. Setup the environment in the shell to LD_PRELOAD libClientSocket.so and
libAlpineSupport.so and set LD_LIBRARY_PATH appropriately. Then run
ping 10.11.12.13.
2. Run ping 10.11.12.13 on a different machine that the one hosting the
Alpine stack.
3. If the host Linux box has two interfaces, then dedicate one to Alpine
and use the other one for Linux. Don't assign any IP address (on Linux)
to the interface assigned to Alpine.
Q. I cannot run "ping". It exits with an error message like:
"error while loading shared libraries: libClientSocket.so".
A. This happens because "ping" is a setuid program. You should be able to
get it to work by "su"ing before running "ping". Or you could try to
use the "ping" program supplied with Alpine4Linux distribution under
the src/sbin/ping directory. It is not setuid and should with regular
user permissions.
Q. I get an error like "Error writing select_resp" when I kill a client
program. Whats up with that ?
A. Its not an error although it looks like one. It happens if a client program
was sleeping in tsleep() but exited before the sleeping system call had
completed. For reasons that are too arcane to go into, we wake up the
sleeping system call as if the process received a SIGKILL. The system
call returns to alpine_server and it tries to write the response back.
But since the client program is already dead, it gets an error from
write() causing this error message. At that point it cleans up state
associated with the client program (closing open sockets, freeing memory
etc).
The short answer is that it is not an error.
Q. I cannot run the native "route" program against Alpine4Linux. How do I
setup routes in the FreeBSD stack ?
A. I had a lot of difficulty making the Linux "route" command work with
the FreeBSD stack. So I had to port the FreeBSD route command for
Alpine4Linux. You can find it under the $(ALPINE_ROOT)/src/sbin/route
directory.
> # Setup the environment correctly
> export LD_PRELOAD="libClientSocket.so libAlpineSupport.so"
> export LD_LIBRARY_PATH="$ALPINE_ROOT/src/client_socket:\
$ALPINE_ROOT/src/alpine_support"
> # See a route to a particular destination IP
> $(ALPINE_ROOT)/src/sbin/route/route -n get 10.11.12.13
> # Now add the default route
> $(ALPINE_ROOT)/src/sbin/route/route add -- -net 0.0.0.0/0 10.11.12.1
Q. The "sbin/route" program supplied with Alpine4Linux exits with the
following error: "route: writing to routing socket: No such process"
A. This happens when you query a the routing table with a destination,
for which a route does not exist. I guess I am messing up when translating
the errnos between FreeBSD and Linux giving rise to the weird error
message.
> $(ALPINE_ROOT)/src/sbin/route/route -n get
Q. I cannot flush the routes from the FreeBSD stack.
A. Sorry. Alpine4Linux does not support sysctl() yet which is needed to
flush the routes. A lame workaround is to restart the alpine_server.
Q. vsftpd dies with an error "500 OOPS: accept".
A. I have seen this with vsftpd-1.1.1, and I believe it is an error in
vsftpd; it does not handle select() timeout correctly when listening
for new connections. It has been fixed in vsftpd-1.1.2 and beyond.
Q. The ftp client talking to vsftpd-1.1.3 transfers data properly but
the ftp server sends a message "426: Failure writing network stream".
A. This is a bug the vsftpd-1.1.3 and is promised to be fixed in a later
release according to chris@scary.beasts.org. The bug is that after
writing the file out using sendfile(), the code checks the errno value
irrespective of whether an error had occurred.
Q. I cannot use the "active" mode ftp with vsftpd. Whats up with that?
A. See limitations about how Alpine4Linux does not support sending file
descriptors between processes.
Q. I cannot make the standard Linux "ftp" client use the Alpine stack. It
simply hangs after connecting to the remote ftp server.
A. Yeah. It sucks. My speculation is that ftp is using getc/putc
to read/write to the socket and this is going to glibc instead of
Alpine4Linux. I have not spent any reasonable amount of time
diagnosing it so I could be completely wrong.
No comments:
Post a Comment