Thursday, July 07, 2011

Alpine user space TCP/IP missing from the Net.

ALPINE:  Application-L evel Protocol Infrastructure for Network Experimentation  

UPDATE July 20, 2011:
I found my Alpine code, but it's only the Alpine4Linux. 

The file can be downloaded from :
I'd like to put these up on source forge or at least something like where the code can no longer get lost. 

In a reply to: My notes on TCP/IP stacks.
chakster said...
Any suggestions on where I can lay my hands on the ALPINE src code and their documentation. The project is conspicuously missing on the web.

As far as I can tell Alpine and Alpine4Linux were user-space ports of the FreeBSD 4.8 TCP networking stack

The URL's were
Unfortunately, these doesn't exist any more.

I found a paper on it:

If anyone has the code, please let me know, send me a copy if possible.
Meanwhile if I ever get my dead server I will try to recover my copy off the disks.
I would really love to have this code. 

List:       linux-net
Subject:    Re: Is it possible to run TCP/IP stack in user space?
From:       John Heffner 
Date:       2006-11-09 5:29:14
Subramonia Pillai wrote:
> Hi,
> I have one doubt. Is it possible to run a third party
> TCP/IP stack as application over a native linux
> kernel. Please give some pointer how it will be done?

Maybe, with some difficulty.

I'm aware of a few things that did this over BSD, but they're really out 
of date.  The Alpine project pulled the BSD TCP stack out into a 
userspace library: 
<>.  The 
FoxNet project wrote a full TCP/IP stack from the ground up in SML: 
<>.  IIRC, they both had similar 
hack layers for running on top of a unix kernel, using pcap, firewall 
filters and raw sockets.

Harvested from, looks like appeared in 2003 then vanished in 2005.


Neelkanth Natu (

Alpine4Linux is a userlevel FreeBSD 4.8 networking stack running on top of a stock Linux kernel. It is an implementation of an idea that I came across in a paper by David Ely, Stefan Savage and David Wetherall. I would like to point out right away that the Alpine4Linux design is completely different from the implementation described in the paper. In particular Alpine4Linux has a single FreeBSD networking stack that is shared by multiple client processes.

Alpine4Linux consists of a userland server program that runs the FreeBSD kernel code as well as the unmodified networking stack.  Client programs use the Alpine stack by setting the LD_PRELOAD environment variable to link to libraries, that intercept socket related system calls. These intercepted system calls are routed to the Alpine server over a TCP connection established on the loopback interface.

Alpine4Linux goes great lengths to ensure that its behavior is identical to that of a FreeBSD kernel as far as networking is concerned. Thus, in addition to the unmodified FreeBSD stack. Alpine4Linux also has unmodified socket layer code, file descriptor code, tsleep and wakeup for e.g. It also has a rich client-side library that supports almost all socket functions as well as functions like fork() that are commonly used by server programs.

Alpine4Linux Server in Action

Here is a output of the Alpine4Linux server when it starts up. The server initializes the kernel when it starts up which is the reason for the FreeBSD copyright messages.

[root@localhost scripts]# bash

Modifying INPUT and FORWARD chains to drop packets destined to

Listening on!
Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD 4.8-RELEASE #4: Fri May 30 02:29:53 PDT 2003
Opening tap0
Setting hwaddr for tap0 device to: 0:4:76:ec:2a:f0
Setting IP address of tap0 device to:

Alpine4Linux uses the "tap0" pseudo-device within the FreeBSD kernel to do raw packet I/O. The "tap0" interface is mapped to a real Linux device (e.g. "eth0") on which the packet I/O really happens. The server also sets the hardware address of the "tap0" pseudo-device to match the hardware address of the real device "eth0". This allows the FreeBSD stack to reply to arp requests for its IP address with a valid hardware address. Note that we change iptables ruleset so that the Linux kernel will ignore packets destined to the Alpine stack.

Alpine4Linux Clients in Action

We can run networking utilities like "ifconfig" and "route" against the Alpine4Linux stack. Note that the "ifconfig" program is the stock Linux program. Also note that we are using a wrapper shell script to execute the programs. The shell script sets up the LD_PRELOAD and LD_LIBRARY_PATH variables before launching the program.

[neel@localhost scripts]$ bash /sbin/ifconfig tap0
tap0      Link encap:Ethernet  HWaddr 00:04:76:EC:2A:F0
          inet addr:  Bcast:  Mask:

[neel@localhost scripts]$ bash ../sbin/route/route -- add -interface
add net gateway
[neel@localhost scripts]$ bash ../sbin/route/route -- get
   route to:
  interface: tap0
 recvpipe  sendpipe  ssthresh  rtt,msec    rttvar  hopcount      mtu     expire
       0         0         0         0         0         0      1500       -12

I have also tested client programs like nmap, telnet and server programs like vsftpd-1.1.3 successfully against Alpine4Linux.

Source Code

You can download the source code here. Untar the tarball using 'tar xvzf alpine4linux.tar.gz' and look for the file docs/running.txt to start playing with Alpine4Linux.


Get Alpine4Linux up and running quick: running.txt

A description of how Alpine4Linux pretends to be the FreeBSD kernel: alpine4linux.txt

FAQ: faq.txt

Limitations of Alpine4Linux: limitations.txt

Differences between Alpine4Linux and the original Alpine implementation: differences_from_alpine4bsd.txt


David Ely, Stefan Savage, David Wetherall for Alpine
Perforce source control system for their free 2-client license
Shri and Ravi for tolerating me when I am babbling


Alpine4Linux is a userlevel FreeBSD 4.8 networking stack running on top
of a stock Linux kernel. The original idea is attributed to [1]. However
I must point out the Alpine4Linux is *not* a port of the original Alpine
(referred to as Alpine4BSD henceforth). In fact there is not a single line 
of code common to the two implementations.

Alpine4Linux has two components:
1. A daemon (alpine_server), that runs the FreeBSD stack code and does
   network I/O on behalf of processes wishing to use the FreeBSD stack.
2. Shared libraries ( and that
   hijack networking related system calls and divert them to the

Why Alpine4Linux:

I did this project because I was fascinated by the idea of running
kernel components in a userlevel process. But seriously, I don't know why 
anyone would want to run a FreeBSD stack in userspace on a Linux box. 

The authors of the original Alpine4BSD paper cite better debugging 
and faster compile-test cycles when doing network protocol development.
That seems to be as good a reason as any.

Supported versions:

The FreeBSD stack is the 4.8-RELEASE version (downloaded on April 8 2003).

The only Linux dependency that the code has is that it support PF_PACKET
family of sockets and it should support makecontext() and swapcontext(). 
Other than that it should run on any Linux distribution.
uname -a on my Linux box:
Linux localhost 2.4.20-13.8 #1 Mon May 12 12:20:54 EDT 2003 i686 i686 i386 GNU/Linux

How does it work:

The alpine_server is a Linux program that acts like a FreeBSD kernel as
far as the networking stack is concerned. Just like the FreeBSD kernel it
provides client programs with a socket layer. It also does network I/O 
on behalf of client programs. But that is where the similarity ends. 

A kernel presents a system call interface to client programs. The
alpine_server presents a RPC interface. RPC here simply means that 
it listens for requests over the network. 

For e.g. if the client program tries to open a socket(), a message 
will be sent to the alpine_server. The message will indicate the 
type of the request (REQ_SOCKET) and its parameters (AF_INET,SOCK_STREAM).
The response will contain the type of the response (RESP_SOCKET) and
the return value (socket_fd, errno).

Client programs link with 2 shared libraries and using LD_PRELOAD and LD_LIBRARY_PATH environment
variables. These libraries "intercept" the socket related functions before
they can be processed conventionally by the Linux libc. Instead each
socket related system call (e.g. socket(), bind(), connect()) is transformed
into a message to the alpine_server.

The alpine_server and its clients communicate over a standard TCP socket
bound to


An unmodified networking stack:

The sys/net, sys/netinet and certain files under sys/kern must be from
the stock FreeBSD-4.8 release. This requirement was mostly satisfied.
I had to make 3 changes to work around some differences between Linux 
and FreeBSD. The changes are trivial and do not change functionality. 
See Appendix A for more details on these changes.

Ability to use unmodified Linux binaries:

It should be possible to simply define the LD_PRELOAD and LD_LIBRARY_PATH
variables and run any dynamically linked Linux networking application
against the FreeBSD stack.

E.g. It is possible to configure the FreeBSD stack interfaces using the
stock 'ifconfig' on Linux. I have also run 'telnet', 'nmap' and 'ping' 
against the alpine_server with no problems. Unfortunately it was too
difficult to use the Linux 'route' command against the FreeBSD stack.
Alpine4Linux provides the 'route' program from FreeBSD, ported to Linux,
that can be used to configure routing in the FreeBSD stack.

Reuse as much of the FreeBSD kernel code as possible:

This requirement is subjective but I think Alpine4Linux utilizes
a *lot* of unmodified FreeBSD kernel code. In fact the alpine_server 
defines only two non-trivial functions that are required by the kernel:
mi_switch() and scheduler().

scheduler() runs the main select() loop in the alpine_server.
mi_switch() deals with switching the FreeBSD kernel execution context.
These functions are described in detail later in this document.

Alpine4BSD uses unmodified sysinit, timeouts, tsleep() and wakeup(),
descriptor management for e.g.


Sending and receiving packets:

The alpine_server is invoked with the name of the interface (on the host
OS) that it uses to send/receive packets. The IP address and subnet that
are used by the FreeBSD stack are also specified on the command line.

E.g. ./alpine_server eth0
This tells the alpine_server to use the "eth0" interface on Linux to
send/receive packets. It also assigns as the IP address
of the FreeBSD stack.

The alpine_server first opens a socket of family PF_PACKET. This is the
recommended way to do raw packet I/O on Linux. A BPF program is compiled,
so that only packets destined for the FreeBSD stack are injected into the
stack. The BPF filter expression is "host ".
Lets call this file descriptor the 'linux_pcap_fd'.

Next we open the "tap" pseudo-device in the FreeBSD stack. This device
presents an Ethernet device interface to the FreeBSD stack.
On the other side the "tap" device returns an 'fd' that can be read and
written to inject raw ethernet packets into the FreeBSD stack. Lets call
this file descriptor the 'freebsd_tap_fd'.

The alpine_server now configures this "tap" device by setting its MAC
address to the MAC address of the interface specified on the command line.
It also sets the IP address of the "tap" device to that specified on the
command line.

Now the job of the alpine_server is simply to read a packet from
'linux_pcap_fd'; run the packet through the BPF filter, and write
the packet to 'freebsd_tap_fd'. In the other direction it reads from
'freebsd_tap_fd' and writes to 'linux_pcap_fd'.

Simulating interrupts:

The alpine_server puts the 'linux_pcap_fd' in its select() read fdset. 
Whenever a packet arrives at the interface, select() returns and 
the packet can be read, filtered and injected into the FreeBSD stack. 

Alpine4Linux acts like a true interrupt driven stack because we
inject packets into the FreeBSD stack as and when we get them.

Software interrupts:

The alpine_server has only one thread of control running at any point in
time. There is no need to lock data structures because this thread
of control cannot be preempted; it has to voluntarily relinquish CPU by
calling mi_switch(). Thus all the spl* functions are no-ops in Alpine4Linux.

setsoftnet() is also a no-op in Alpine because we run the netisrs
periodically. In Alpine4Linux this happens at every tick (1/HZ secs).

The function do_netisrs() defined in kern/kern_netisr.c is called periodically
by the alpine_server. This function calls all the netisrs ready to run,
and gives them the opportunity to drain packets from their packet queues.


Alpine4Linux initializes the kernel data structures as if the kernel had
booted itself. The alpine_server contains main() that is the entry point
into the program. main() in turn calls init386() followed by mi_startup().

init386() was rewritten to only initialize the tunable variables in the
kernel like "hz" or "tick". It also initializes physical memory dependent
variables like "maxusers" and "maxproc". Alpine4Linux makes the FreeBSD
kernel believe that it is running on a machine with 1Gbytes of physical

This is the stock mi_startup() from the FreeBSD kernel, since we support
sysinit in This function does not return and control
ends up in the scheduler() function. Alpine4Linux defines the scheduler()
function in alpine_server. Eventually control lands in the main select()
loop defined in sched_main_loop().

Timer management:

Timer management in Alpine4Linux is very simple. In the main select() loop,
we call hardclock() every 10 msec (this interval is based on kern.hz).
If there is any event in the current timer wheel bucket, softclock() is 
called from hardclock(). At that point the stock FreeBSD code is used to deal
with timeout events. slowtimo() and fasttimo() are indirectly called using 
this mechanism.

tsleep() and wakeup():

Alpine4Linux uses the stock tsleep() and wakeup() functions from FreeBSD
without any modifications. The blocking behavior of a process in the kernel
is implemented by mi_switch() that is defined outside the kernel.

Multiple execution contexts in the stack:

The alpine_server provides networking services to multiple clients at the
same time. It is thus imperative that the alpine_server not block in the
kernel. This is the same constraint that the FreeBSD kernel itself operates
under. Anytime a client process does an action that causes it to block (e.g.
a blocking read() on a socket), the alpine_server must store the execution
context and switch to another client process that is ready to run. If there
are no client processes ready to run, the alpine_server blocks in select().
The select() loop of alpine_server is analogous to the idle loop of a Unix

The alpine_server provides multiple execution contexts (one for each client),
using the makecontext(3) function available in Linux. It switches between
execution contexts in the FreeBSD kernel using swapcontext(3).

The alpine_server itself executes in a 'ucontext_t' that is accessible as
a global variable (sched_thread->ut_ctx). The alpine_server (and hence 
the FreeBSD stack) executes in this context for system level events like 
timeouts, network I/O etc. 

The alpine_server executes in a 'ucontext_t' associated with a client process
whenever it is executing code in the FreeBSD kernel on behalf of the client
process. For e.g. if the client process sends a messages to the alpine_server
to read() from a socket, the alpine_server first creates a 'ucontext_t' and
switches execution to the newly created context. If all goes well and there
is data to be read, we will reply back to the client_process; the newly
created 'ucontext_t' will be destroyed and control passes back to the
main 'sched_thread' context. If there is not enough data to be read, then
the ucontext_t will need to block in tsleep() and it will call mi_switch().
mi_switch() does a swapcontext() to the main 'sched_thread'. The sched_thread 
either idles on select() or does a swapcontext() to process a request 
from another client.

The alpine_server also has to take care of "woken-up" execution contexts.
For e.g. Consider an execution context that was put to sleep because there
was not enough data to satisfy a read(). When data arrives on that socket,
that execution context becomes runnable (i.e. p->p_stat == SRUN). We check
for such "woken-up" processes just before the sched_thread sleeps in select().
It traverses all the ucontexts that are sleeping state *but* their proc
structure is runnable i.e.(ut->ut_state==UTS_SLEEPING && p->p_stat==SRUN).
if such a ucontext is found it does a swapcontext() to it. The blocked
ucontext resumes execution after the mi_switch() statement in tsleep() just
like it would in a stock FreeBSD kernel.

Interaction with the Linux kernel:

Alpine4Linux is a pure userlevel process and requires no Linux kernel
modifications. But Alpine4Linux is handing out file descriptors to
client programs (fd = socket()); it needs to ensure it does not step on
the Linux kernel's toes when it does so.

Therefore we need to map file descriptors between the FreeBSD stack and the
host OS. To see why we need this, consider a case where an application
issues a socket() system call. This call is intercepted by the libClientSocket
library and a file descriptor is assigned to the newly created socket by
the FreeBSD stack. Lets call this file descriptor the 'alpine_fd'. We need
to ensure that the value we return to the client application is an fd
that is not already been allocated and will not be allocated in the future.
Hence we open("/dev/null") on the host OS and create a mapping between
linux_fd and alpine_fd. The fd that is returned to the client from the
socket() system call is the linux_fd.

When the client comes back to do read() or write() with the linux_fd, we
will map that fd to the alpine_fd and use it in the FreeBSD stack.

Alpine4Linux compared to Alpine4BSD:

Look at the file "differences_from_alpine4bsd.txt" under the docs/ directory
for salient differences between Alpine4Linux and Alpine4BSD.


I have not done any performance measurements for Alpine4Linux, because I
am confident that its performance sucks! Since Alpine4Linux does message
passing between the client program and alpine_server there are a lot of
copies of when reading or writing data.

Future work:

Support IPv6, IPSEC etc.

[1] Alpine: A User-Level Infrastructure for Network Protocol Development
    David Ely, Stefan Savage, David Wetherall

Appendix A:

The following are the descriptions of the 3 changes I had to make in the
FreeBSD stack to make it work on Linux. All changes are trivial and do
not affect functionality.

printf on Linux does not have the %D modifier

ifconfig on Linux do not zero out sin_zero in sockaddr_in when doing
when binding to that IP address, because ifa_ifwithaddr() compares the 
entire 16 bytes; but there is garbage in the last 8 bytes of ifa->ifa_addr. 
The fix was to zero out sin_zero of ia->ia_addr, ia->ia_dstaddr and 
ia->ia_broadaddr in in_ifinit().

We map the Linux SIOCSIFHWADDR to the FreeBSD SIOCSIFLLADDR. However 
Linux does not define the sa_len member in its sockaddr structure.
We should not return EINVAL if if the 'sa_len' does not match 
'sdl->sdl_alen' in if_setlladdr().

This file describes the limitations of Alpine4Linux.


Alpine4Linux cannot pass file descriptors from one process to another. Some
programs (e.g. vsftpd) use this feature; a privileged process binds
a socket to a privileged port and then transfers this fd to a less
privileged process to do the actual data transfer. Therefore vsftpd 
"active" mode does not work with Alpine4Linux.


This file describes the differences between the original Alpine
implementation and Alpine4Linux. The original Alpine is referred
to as Alpine4BSD henceforth.


Alpine4Linux needs a unique IP address separate from the host OS.
Alpine4BSD shares the IP address already assigned to the host OS.
I chose this approach because:
1. It is simpler (I did not have to write code to share the port-space
   with the host kernel)
2. Alpine4Linux will be used in a research environment where assigning the
   host OS an additional IP address should not be difficult.
As long as the host OS does not send RSTs or ICMP unreach messages to
the sender, we should be fine. Such a "blackhole" behavior can be
configured on a Linux box using iptables. Alpine4Linux provides
a helper script - "" - that configures iptables
before starting alpine_server. It also cleans up when alpine_server

Alpine4Linux support fork()! 
This is probably the biggest difference from Alpine4BSD.

Alpine4BSD uses a faux-ethernet device that was newly written to inject
packets into and get packets from the FreeBSD stack. I leveraged the
"tap" pseudo-device already present in the FreeBSD stack to achieve 
identical functionality. No code change were made to the "tap" driver.

Alpine4BSD requires support for BPF devices to be compiled in the
host OS. Alpine4Linux requires support for PF_PACKET sockets in the
host OS.

Alpine4BSD simulates interrupts by polling pcap once every 1ms by using
SIGALRM. We don't have to do that since we put the 'linux_pcap_fd' in
select(), so alpine_server is woken up every time a packet is available
to read on the pcap_fd. In this sense Alpine4Linux is a true interrupt
driven stack.

Q. I don't want the alpine_server to listen on How do I tell
   it to listen for requests on another address?

A. Define the environment variables ALPINE_SERVER_LISTEN_ADDR and 
   ALPINE_SERVER_LISTEN_PORT appropriately. Make sure that the client programs
   also execute in an enviroment with the same variables defined.

Q. I can't ping the Alpine4Linux IP address from the same machine that it
   is running on.

A. Let me describe the setup first, the problem next and the solution

   The setup:
   Consider that the native linux machine has an interface eth0 with an IP
   address We run Alpine4Linux on the same interface with an IP
   address (./alpine_server eth0
   The problem:
   Now if we try to ping from the Linux box to we won't be able 
   to "see" the Alpine stack. This is because the ARP request from the Linux 
   box is thrown away by the Alpine stack because it has a source hardware 
   address that Alpine considers to be its own. It believes that it is 
   seeing an echo of its own ARP request and the packet is discarded.
   The solution:
   1. Setup the environment in the shell to LD_PRELOAD and and set LD_LIBRARY_PATH appropriately. Then run
   2. Run ping on a different machine that the one hosting the
      Alpine stack.
   3. If the host Linux box has two interfaces, then dedicate one to Alpine
      and use the other one for Linux. Don't assign any IP address (on Linux)
      to the interface assigned to Alpine.

Q. I cannot run "ping". It exits with an error message like:
   "error while loading shared libraries:".
A. This happens because "ping" is a setuid program. You should be able to
   get it to work by "su"ing before running "ping". Or you could try to
   use the "ping" program supplied with Alpine4Linux distribution under
   the src/sbin/ping directory. It is not setuid and should with regular
   user permissions.

Q. I get an error like "Error writing select_resp" when I kill a client
   program. Whats up with that ?

A. Its not an error although it looks like one. It happens if a client program
   was sleeping in tsleep() but exited before the sleeping system call had
   completed. For reasons that are too arcane to go into, we wake up the
   sleeping system call as if the process received a SIGKILL. The system
   call returns to alpine_server and it tries to write the response back.
   But since the client program is already dead, it gets an error from
   write() causing this error message. At that point it cleans up state
   associated with the client program (closing open sockets, freeing memory 
   The short answer is that it is not an error.

Q. I cannot run the native "route" program against Alpine4Linux. How do I
   setup routes in the FreeBSD stack ?

A. I had a lot of difficulty making the Linux "route" command work with
   the FreeBSD stack. So I had to port the FreeBSD route command for
   Alpine4Linux. You can find it under the $(ALPINE_ROOT)/src/sbin/route
   > # Setup the environment correctly
   > export LD_PRELOAD=""
   > export LD_LIBRARY_PATH="$ALPINE_ROOT/src/client_socket:\
   > # See a route to a particular destination IP
   > $(ALPINE_ROOT)/src/sbin/route/route -n get
   > # Now add the default route
   > $(ALPINE_ROOT)/src/sbin/route/route add -- -net

Q. The "sbin/route" program supplied with Alpine4Linux exits with the
   following error: "route: writing to routing socket: No such process"
A. This happens when you query a the routing table with a destination,
   for which a route does not exist. I guess I am messing up when translating
   the errnos between FreeBSD and Linux giving rise to the weird error
   > $(ALPINE_ROOT)/src/sbin/route/route -n get

Q. I cannot flush the routes from the FreeBSD stack.
A. Sorry. Alpine4Linux does not support sysctl() yet which is needed to
   flush the routes. A lame workaround is to restart the alpine_server.

Q. vsftpd dies with an error "500 OOPS: accept".
A. I have seen this with vsftpd-1.1.1, and I believe it is an error in
   vsftpd; it does not handle select() timeout correctly when listening
   for new connections. It has been fixed in vsftpd-1.1.2 and beyond.

Q. The ftp client talking to vsftpd-1.1.3 transfers data properly but
   the ftp server sends a message "426: Failure writing network stream".
A. This is a bug the vsftpd-1.1.3 and is promised to be fixed in a later
   release according to The bug is that after
   writing the file out using sendfile(), the code checks the errno value
   irrespective of whether an error had occurred.

Q. I cannot use the "active" mode ftp with vsftpd. Whats up with that?
A. See limitations about how Alpine4Linux does not support sending file
   descriptors between processes.

Q. I cannot make the standard Linux "ftp" client use the Alpine stack. It
   simply hangs after connecting to the remote ftp server.
A. Yeah. It sucks. My speculation is that ftp is using getc/putc
   to read/write to the socket and this is going to glibc instead of
   Alpine4Linux. I have not spent any reasonable amount of time 
   diagnosing it so I could be completely wrong.

No comments: