Alpine user space TCP/IP missing from the Net.

ALPINE:  Application-L evel Protocol Infrastructure for Network Experimentation  

UPDATE July 20, 2011:

I found my Alpine code, but it's only the Alpine4Linux. 

The file can be downloaded from : http://www.dnull.com/Alpine/

I'd like to put these up on source forge or at least something like box.net where the code can no longer get lost. 

In a reply to: My notes on TCP/IP stacks.

chakster said...

Any suggestions on where I can lay my hands on the ALPINE src code and their documentation. The project is conspicuously missing on the web.

As far as I can tell Alpine and Alpine4Linux were user-space ports of the FreeBSD 4.8 TCP networking stack

The URL's were

http://alpine.cs.washington.edu/
 http://www.vzavenue.net/~neelnatu/alpine4linux/.
Unfortunately, these doesn't exist any more.

I found a paper on it:
http://www.usenix.org/event/usits01/full_papers/ely/alpineUs.pdf


If anyone has the code, please let me know, send me a copy if possible.
Meanwhile if I ever get my dead server I will try to recover my copy off the disks.
I would really love to have this code. 

---

List:       linux-net
Subject:    Re: Is it possible to run TCP/IP stack in user space?
From:       John Heffner 
Date:       2006-11-09 5:29:14
Subramonia Pillai wrote:
> Hi,
> 
> I have one doubt. Is it possible to run a third party
> TCP/IP stack as application over a native linux
> kernel. Please give some pointer how it will be done?

Maybe, with some difficulty.

I'm aware of a few things that did this over BSD, but they're really out 
of date.  The Alpine project pulled the BSD TCP stack out into a 
userspace library: 
<http://sosp16.cs.washington.edu/homes/djw/papers/Usits01.pdf>.  The 
FoxNet project wrote a full TCP/IP stack from the ground up in SML: 
<http://www.cs.cmu.edu/~fox/foxnet.html>.  IIRC, they both had similar 
hack layers for running on top of a unix kernel, using pcap, firewall 
filters and raw sockets.

---

Harvested from Archive.org, looks like appeared in 2003 then vanished in 2005.

Alpine4Linux

Neelkanth Natu (neelnatu@yahoo.com)

Alpine4Linux is a userlevel FreeBSD 4.8 networking stack running on top of a stock Linux kernel. It is an implementation of an idea that I came across in a paper by David Ely, Stefan Savage and David Wetherall. I would like to point out right away that the Alpine4Linux design is completely different from the implementation described in the paper. In particular Alpine4Linux has a single FreeBSD networking stack that is shared by multiple client processes.

Alpine4Linux consists of a userland server program that runs the FreeBSD kernel code as well as the unmodified networking stack.  Client programs use the Alpine stack by setting the LD_PRELOAD environment variable to link to libraries, that intercept socket related system calls. These intercepted system calls are routed to the Alpine server over a TCP connection established on the loopback interface.

Alpine4Linux goes great lengths to ensure that its behavior is identical to that of a FreeBSD kernel as far as networking is concerned. Thus, in addition to the unmodified FreeBSD stack. Alpine4Linux also has unmodified socket layer code, file descriptor code, tsleep and wakeup for e.g. It also has a rich client-side library that supports almost all socket functions as well as functions like fork() that are commonly used by server programs.

Alpine4Linux Server in Action

Here is a output of the Alpine4Linux server when it starts up. The server initializes the kernel when it starts up which is the reason for the FreeBSD copyright messages.

[root@localhost scripts]# bash run_alpine_server.sh

Modifying INPUT and FORWARD chains to drop packets destined to 10.11.12.13

Listening on 127.0.0.1:8475!
Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD 4.8-RELEASE #4: Fri May 30 02:29:53 PDT 2003
    neel@localhost:/home/neel/av/main/bsd/src/sys/compile/NINT
Opening tap0
Setting hwaddr for tap0 device to: 0:4:76:ec:2a:f0
Setting IP address of tap0 device to: 10.11.12.13

Alpine4Linux uses the "tap0" pseudo-device within the FreeBSD kernel to do raw packet I/O. The "tap0" interface is mapped to a real Linux device (e.g. "eth0") on which the packet I/O really happens. The server also sets the hardware address of the "tap0" pseudo-device to match the hardware address of the real device "eth0". This allows the FreeBSD stack to reply to arp requests for its IP address with a valid hardware address. Note that we change iptables ruleset so that the Linux kernel will ignore packets destined to the Alpine stack.

Alpine4Linux Clients in Action

We can run networking utilities like "ifconfig" and "route" against the Alpine4Linux stack. Note that the "ifconfig" program is the stock Linux program. Also note that we are using a wrapper shell script to execute the programs. The shell script sets up the LD_PRELOAD and LD_LIBRARY_PATH variables before launching the program.

[neel@localhost scripts]$ bash run_alpine_client.sh /sbin/ifconfig tap0
tap0      Link encap:Ethernet  HWaddr 00:04:76:EC:2A:F0
          inet addr:10.11.12.13  Bcast:10.11.12.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1


[neel@localhost scripts]$ bash run_alpine_client.sh ../sbin/route/route -- add 10.11.12.0/24 -interface 10.11.12.13
add net 10.11.12.0: gateway 10.11.12.13
[neel@localhost scripts]$ bash run_alpine_client.sh ../sbin/route/route -- get 10.11.12.1
   route to: 10.11.12.1
destination: 10.11.12.0
       mask: 255.255.255.0
  interface: tap0
      flags:
 recvpipe  sendpipe  ssthresh  rtt,msec    rttvar  hopcount      mtu     expire
       0         0         0         0         0         0      1500       -12

I have also tested client programs like nmap, telnet and server programs like vsftpd-1.1.3 successfully against Alpine4Linux.

Source Code

You can download the source code here. Untar the tarball using 'tar xvzf alpine4linux.tar.gz' and look for the file docs/running.txt to start playing with Alpine4Linux.

Documentation

Get Alpine4Linux up and running quick: running.txt

A description of how Alpine4Linux pretends to be the FreeBSD kernel: alpine4linux.txt

FAQ: faq.txt

Limitations of Alpine4Linux: limitations.txt

Differences between Alpine4Linux and the original Alpine implementation: differences_from_alpine4bsd.txt

Acknowledgements

David Ely, Stefan Savage, David Wetherall for Alpine
Perforce source control system for their free 2-client license
Shri and Ravi for tolerating me when I am babbling

---

Introduction:

Alpine4Linux is a userlevel FreeBSD 4.8 networking stack running on top
of a stock Linux kernel. The original idea is attributed to [1]. However
I must point out the Alpine4Linux is *not* a port of the original Alpine
(referred to as Alpine4BSD henceforth). In fact there is not a single line 
of code common to the two implementations.

Alpine4Linux has two components:
1. A daemon (alpine_server), that runs the FreeBSD stack code and does
   network I/O on behalf of processes wishing to use the FreeBSD stack.
2. Shared libraries (libClientSocket.so and libAlpineSupport.so) that
   hijack networking related system calls and divert them to the
   alpine_server. 

Why Alpine4Linux:

I did this project because I was fascinated by the idea of running
kernel components in a userlevel process. But seriously, I don't know why 
anyone would want to run a FreeBSD stack in userspace on a Linux box. 

The authors of the original Alpine4BSD paper cite better debugging 
and faster compile-test cycles when doing network protocol development.
That seems to be as good a reason as any.

Supported versions:

The FreeBSD stack is the 4.8-RELEASE version (downloaded on April 8 2003).

The only Linux dependency that the code has is that it support PF_PACKET
family of sockets and it should support makecontext() and swapcontext(). 
Other than that it should run on any Linux distribution.
uname -a on my Linux box:
Linux localhost 2.4.20-13.8 #1 Mon May 12 12:20:54 EDT 2003 i686 i686 i386 GNU/Linux

How does it work:

The alpine_server is a Linux program that acts like a FreeBSD kernel as
far as the networking stack is concerned. Just like the FreeBSD kernel it
provides client programs with a socket layer. It also does network I/O 
on behalf of client programs. But that is where the similarity ends. 

A kernel presents a system call interface to client programs. The
alpine_server presents a RPC interface. RPC here simply means that 
it listens for requests over the network. 

For e.g. if the client program tries to open a socket(), a message 
will be sent to the alpine_server. The message will indicate the 
type of the request (REQ_SOCKET) and its parameters (AF_INET,SOCK_STREAM).
The response will contain the type of the response (RESP_SOCKET) and
the return value (socket_fd, errno).

Client programs link with 2 shared libraries libClientSocket.so and
libAlpineSupport.so using LD_PRELOAD and LD_LIBRARY_PATH environment
variables. These libraries "intercept" the socket related functions before
they can be processed conventionally by the Linux libc. Instead each
socket related system call (e.g. socket(), bind(), connect()) is transformed
into a message to the alpine_server.

The alpine_server and its clients communicate over a standard TCP socket
bound to 127.0.0.1.

Requirements:

An unmodified networking stack:

The sys/net, sys/netinet and certain files under sys/kern must be from
the stock FreeBSD-4.8 release. This requirement was mostly satisfied.
I had to make 3 changes to work around some differences between Linux 
and FreeBSD. The changes are trivial and do not change functionality. 
See Appendix A for more details on these changes.

Ability to use unmodified Linux binaries:

It should be possible to simply define the LD_PRELOAD and LD_LIBRARY_PATH
variables and run any dynamically linked Linux networking application
against the FreeBSD stack.

E.g. It is possible to configure the FreeBSD stack interfaces using the
stock 'ifconfig' on Linux. I have also run 'telnet', 'nmap' and 'ping' 
against the alpine_server with no problems. Unfortunately it was too
difficult to use the Linux 'route' command against the FreeBSD stack.
Alpine4Linux provides the 'route' program from FreeBSD, ported to Linux,
that can be used to configure routing in the FreeBSD stack.

Reuse as much of the FreeBSD kernel code as possible:

This requirement is subjective but I think Alpine4Linux utilizes
a *lot* of unmodified FreeBSD kernel code. In fact the alpine_server 
defines only two non-trivial functions that are required by the kernel:
mi_switch() and scheduler().

scheduler() runs the main select() loop in the alpine_server.
mi_switch() deals with switching the FreeBSD kernel execution context.
These functions are described in detail later in this document.

Alpine4BSD uses unmodified sysinit, timeouts, tsleep() and wakeup(),
descriptor management for e.g.

Implementation:

Sending and receiving packets:

The alpine_server is invoked with the name of the interface (on the host
OS) that it uses to send/receive packets. The IP address and subnet that
are used by the FreeBSD stack are also specified on the command line.

E.g. ./alpine_server eth0 10.11.12.13 255.255.255.0
This tells the alpine_server to use the "eth0" interface on Linux to
send/receive packets. It also assigns 10.11.12.13/24 as the IP address
of the FreeBSD stack.

The alpine_server first opens a socket of family PF_PACKET. This is the
recommended way to do raw packet I/O on Linux. A BPF program is compiled,
so that only packets destined for the FreeBSD stack are injected into the
stack. The BPF filter expression is "host ".
Lets call this file descriptor the 'linux_pcap_fd'.

Next we open the "tap" pseudo-device in the FreeBSD stack. This device
presents an Ethernet device interface to the FreeBSD stack.
On the other side the "tap" device returns an 'fd' that can be read and
written to inject raw ethernet packets into the FreeBSD stack. Lets call
this file descriptor the 'freebsd_tap_fd'.

The alpine_server now configures this "tap" device by setting its MAC
address to the MAC address of the interface specified on the command line.
It also sets the IP address of the "tap" device to that specified on the
command line.

Now the job of the alpine_server is simply to read a packet from
'linux_pcap_fd'; run the packet through the BPF filter, and write
the packet to 'freebsd_tap_fd'. In the other direction it reads from
'freebsd_tap_fd' and writes to 'linux_pcap_fd'.

Simulating interrupts:

The alpine_server puts the 'linux_pcap_fd' in its select() read fdset. 
Whenever a packet arrives at the interface, select() returns and 
the packet can be read, filtered and injected into the FreeBSD stack. 

Alpine4Linux acts like a true interrupt driven stack because we
inject packets into the FreeBSD stack as and when we get them.

Software interrupts:

The alpine_server has only one thread of control running at any point in
time. There is no need to lock data structures because this thread
of control cannot be preempted; it has to voluntarily relinquish CPU by
calling mi_switch(). Thus all the spl* functions are no-ops in Alpine4Linux.

setsoftnet() is also a no-op in Alpine because we run the netisrs
periodically. In Alpine4Linux this happens at every tick (1/HZ secs).

The function do_netisrs() defined in kern/kern_netisr.c is called periodically
by the alpine_server. This function calls all the netisrs ready to run,
and gives them the opportunity to drain packets from their packet queues.

Initialization:

Alpine4Linux initializes the kernel data structures as if the kernel had
booted itself. The alpine_server contains main() that is the entry point
into the program. main() in turn calls init386() followed by mi_startup().

init386():
init386() was rewritten to only initialize the tunable variables in the
kernel like "hz" or "tick". It also initializes physical memory dependent
variables like "maxusers" and "maxproc". Alpine4Linux makes the FreeBSD
kernel believe that it is running on a machine with 1Gbytes of physical
memory.

mi_startup():
This is the stock mi_startup() from the FreeBSD kernel, since we support
sysinit in libAlpineSys.so. This function does not return and control
ends up in the scheduler() function. Alpine4Linux defines the scheduler()
function in alpine_server. Eventually control lands in the main select()
loop defined in sched_main_loop().

Timer management:

Timeout:
Timer management in Alpine4Linux is very simple. In the main select() loop,
we call hardclock() every 10 msec (this interval is based on kern.hz).
If there is any event in the current timer wheel bucket, softclock() is 
called from hardclock(). At that point the stock FreeBSD code is used to deal
with timeout events. slowtimo() and fasttimo() are indirectly called using 
this mechanism.

tsleep() and wakeup():

Alpine4Linux uses the stock tsleep() and wakeup() functions from FreeBSD
without any modifications. The blocking behavior of a process in the kernel
is implemented by mi_switch() that is defined outside the kernel.

Multiple execution contexts in the stack:

The alpine_server provides networking services to multiple clients at the
same time. It is thus imperative that the alpine_server not block in the
kernel. This is the same constraint that the FreeBSD kernel itself operates
under. Anytime a client process does an action that causes it to block (e.g.
a blocking read() on a socket), the alpine_server must store the execution
context and switch to another client process that is ready to run. If there
are no client processes ready to run, the alpine_server blocks in select().
The select() loop of alpine_server is analogous to the idle loop of a Unix
kernel.

The alpine_server provides multiple execution contexts (one for each client),
using the makecontext(3) function available in Linux. It switches between
execution contexts in the FreeBSD kernel using swapcontext(3).

The alpine_server itself executes in a 'ucontext_t' that is accessible as
a global variable (sched_thread->ut_ctx). The alpine_server (and hence 
the FreeBSD stack) executes in this context for system level events like 
timeouts, network I/O etc. 

The alpine_server executes in a 'ucontext_t' associated with a client process
whenever it is executing code in the FreeBSD kernel on behalf of the client
process. For e.g. if the client process sends a messages to the alpine_server
to read() from a socket, the alpine_server first creates a 'ucontext_t' and
switches execution to the newly created context. If all goes well and there
is data to be read, we will reply back to the client_process; the newly
created 'ucontext_t' will be destroyed and control passes back to the
main 'sched_thread' context. If there is not enough data to be read, then
the ucontext_t will need to block in tsleep() and it will call mi_switch().
mi_switch() does a swapcontext() to the main 'sched_thread'. The sched_thread 
either idles on select() or does a swapcontext() to process a request 
from another client.

The alpine_server also has to take care of "woken-up" execution contexts.
For e.g. Consider an execution context that was put to sleep because there
was not enough data to satisfy a read(). When data arrives on that socket,
that execution context becomes runnable (i.e. p->p_stat == SRUN). We check
for such "woken-up" processes just before the sched_thread sleeps in select().
It traverses all the ucontexts that are sleeping state *but* their proc
structure is runnable i.e.(ut->ut_state==UTS_SLEEPING && p->p_stat==SRUN).
if such a ucontext is found it does a swapcontext() to it. The blocked
ucontext resumes execution after the mi_switch() statement in tsleep() just
like it would in a stock FreeBSD kernel.

Interaction with the Linux kernel:

Alpine4Linux is a pure userlevel process and requires no Linux kernel
modifications. But Alpine4Linux is handing out file descriptors to
client programs (fd = socket()); it needs to ensure it does not step on
the Linux kernel's toes when it does so.

Therefore we need to map file descriptors between the FreeBSD stack and the
host OS. To see why we need this, consider a case where an application
issues a socket() system call. This call is intercepted by the libClientSocket
library and a file descriptor is assigned to the newly created socket by
the FreeBSD stack. Lets call this file descriptor the 'alpine_fd'. We need
to ensure that the value we return to the client application is an fd
that is not already been allocated and will not be allocated in the future.
Hence we open("/dev/null") on the host OS and create a mapping between
linux_fd and alpine_fd. The fd that is returned to the client from the
socket() system call is the linux_fd.

When the client comes back to do read() or write() with the linux_fd, we
will map that fd to the alpine_fd and use it in the FreeBSD stack.

Alpine4Linux compared to Alpine4BSD:

Look at the file "differences_from_alpine4bsd.txt" under the docs/ directory
for salient differences between Alpine4Linux and Alpine4BSD.

Performance:

I have not done any performance measurements for Alpine4Linux, because I
am confident that its performance sucks! Since Alpine4Linux does message
passing between the client program and alpine_server there are a lot of
copies of when reading or writing data.

Future work:

Support IPv6, IPSEC etc.

References:
[1] Alpine: A User-Level Infrastructure for Network Protocol Development
    David Ely, Stefan Savage, David Wetherall
    http://alpine.cs.washington.edu/

Appendix A:

The following are the descriptions of the 3 changes I had to make in the
FreeBSD stack to make it work on Linux. All changes are trivial and do
not affect functionality.

netinet/if_ether.c: 
printf on Linux does not have the %D modifier

netinet/in.c: 
ifconfig on Linux do not zero out sin_zero in sockaddr_in when doing
SIOCSIFADDR, SIOCSIFDSTADDR and SIOCSIFBRDADDR. This causes problems 
when binding to that IP address, because ifa_ifwithaddr() compares the 
entire 16 bytes; but there is garbage in the last 8 bytes of ifa->ifa_addr. 
The fix was to zero out sin_zero of ia->ia_addr, ia->ia_dstaddr and 
ia->ia_broadaddr in in_ifinit().

net/if.c:
We map the Linux SIOCSIFHWADDR to the FreeBSD SIOCSIFLLADDR. However 
Linux does not define the sa_len member in its sockaddr structure.
We should not return EINVAL if if the 'sa_len' does not match 
'sdl->sdl_alen' in if_setlladdr().

---

Introduction:
This file describes the limitations of Alpine4Linux.

Limitations:

Alpine4Linux cannot pass file descriptors from one process to another. Some
programs (e.g. vsftpd) use this feature; a privileged process binds
a socket to a privileged port and then transfers this fd to a less
privileged process to do the actual data transfer. Therefore vsftpd 
"active" mode does not work with Alpine4Linux.

---

Introduction:

This file describes the differences between the original Alpine
implementation and Alpine4Linux. The original Alpine is referred
to as Alpine4BSD henceforth.

Differences:

Alpine4Linux needs a unique IP address separate from the host OS.
Alpine4BSD shares the IP address already assigned to the host OS.
I chose this approach because:
1. It is simpler (I did not have to write code to share the port-space
   with the host kernel)
2. Alpine4Linux will be used in a research environment where assigning the
   host OS an additional IP address should not be difficult.
As long as the host OS does not send RSTs or ICMP unreach messages to
the sender, we should be fine. Such a "blackhole" behavior can be
configured on a Linux box using iptables. Alpine4Linux provides
a helper script - "run_alpine_server.sh" - that configures iptables
before starting alpine_server. It also cleans up when alpine_server
exits.

Alpine4Linux support fork()! 
This is probably the biggest difference from Alpine4BSD.

Alpine4BSD uses a faux-ethernet device that was newly written to inject
packets into and get packets from the FreeBSD stack. I leveraged the
"tap" pseudo-device already present in the FreeBSD stack to achieve 
identical functionality. No code change were made to the "tap" driver.

Alpine4BSD requires support for BPF devices to be compiled in the
host OS. Alpine4Linux requires support for PF_PACKET sockets in the
host OS.

Alpine4BSD simulates interrupts by polling pcap once every 1ms by using
SIGALRM. We don't have to do that since we put the 'linux_pcap_fd' in
select(), so alpine_server is woken up every time a packet is available
to read on the pcap_fd. In this sense Alpine4Linux is a true interrupt
driven stack.

---

Q. I don't want the alpine_server to listen on 127.0.0.1:8475. How do I tell
   it to listen for requests on another address?

A. Define the environment variables ALPINE_SERVER_LISTEN_ADDR and 
   ALPINE_SERVER_LISTEN_PORT appropriately. Make sure that the client programs
   also execute in an enviroment with the same variables defined.

Q. I can't ping the Alpine4Linux IP address from the same machine that it
   is running on.

A. Let me describe the setup first, the problem next and the solution
   last.

   The setup:
   Consider that the native linux machine has an interface eth0 with an IP
   address 10.11.12.1. We run Alpine4Linux on the same interface with an IP
   address 10.11.12.13 (./alpine_server eth0 10.11.12.13 255.255.255.0).
   
   The problem:
   Now if we try to ping from the Linux box to 10.11.12.13 we won't be able 
   to "see" the Alpine stack. This is because the ARP request from the Linux 
   box is thrown away by the Alpine stack because it has a source hardware 
   address that Alpine considers to be its own. It believes that it is 
   seeing an echo of its own ARP request and the packet is discarded.
   
   The solution:
   1. Setup the environment in the shell to LD_PRELOAD libClientSocket.so and
      libAlpineSupport.so and set LD_LIBRARY_PATH appropriately. Then run
      ping 10.11.12.13.
   2. Run ping 10.11.12.13 on a different machine that the one hosting the
      Alpine stack.
   3. If the host Linux box has two interfaces, then dedicate one to Alpine
      and use the other one for Linux. Don't assign any IP address (on Linux)
      to the interface assigned to Alpine.

Q. I cannot run "ping". It exits with an error message like:
   "error while loading shared libraries: libClientSocket.so".
A. This happens because "ping" is a setuid program. You should be able to
   get it to work by "su"ing before running "ping". Or you could try to
   use the "ping" program supplied with Alpine4Linux distribution under
   the src/sbin/ping directory. It is not setuid and should with regular
   user permissions.

Q. I get an error like "Error writing select_resp" when I kill a client
   program. Whats up with that ?

A. Its not an error although it looks like one. It happens if a client program
   was sleeping in tsleep() but exited before the sleeping system call had
   completed. For reasons that are too arcane to go into, we wake up the
   sleeping system call as if the process received a SIGKILL. The system
   call returns to alpine_server and it tries to write the response back.
   But since the client program is already dead, it gets an error from
   write() causing this error message. At that point it cleans up state
   associated with the client program (closing open sockets, freeing memory 
   etc).
   The short answer is that it is not an error.

Q. I cannot run the native "route" program against Alpine4Linux. How do I
   setup routes in the FreeBSD stack ?

A. I had a lot of difficulty making the Linux "route" command work with
   the FreeBSD stack. So I had to port the FreeBSD route command for
   Alpine4Linux. You can find it under the $(ALPINE_ROOT)/src/sbin/route
   directory.
   > # Setup the environment correctly
   > export LD_PRELOAD="libClientSocket.so libAlpineSupport.so"
   > export LD_LIBRARY_PATH="$ALPINE_ROOT/src/client_socket:\
           $ALPINE_ROOT/src/alpine_support"
   > # See a route to a particular destination IP
   > $(ALPINE_ROOT)/src/sbin/route/route -n get 10.11.12.13
   > # Now add the default route
   > $(ALPINE_ROOT)/src/sbin/route/route add -- -net 0.0.0.0/0 10.11.12.1

Q. The "sbin/route" program supplied with Alpine4Linux exits with the
   following error: "route: writing to routing socket: No such process"
A. This happens when you query a the routing table with a destination,
   for which a route does not exist. I guess I am messing up when translating
   the errnos between FreeBSD and Linux giving rise to the weird error
   message.
   > $(ALPINE_ROOT)/src/sbin/route/route -n get

Q. I cannot flush the routes from the FreeBSD stack.
A. Sorry. Alpine4Linux does not support sysctl() yet which is needed to
   flush the routes. A lame workaround is to restart the alpine_server.

Q. vsftpd dies with an error "500 OOPS: accept".
A. I have seen this with vsftpd-1.1.1, and I believe it is an error in
   vsftpd; it does not handle select() timeout correctly when listening
   for new connections. It has been fixed in vsftpd-1.1.2 and beyond.

Q. The ftp client talking to vsftpd-1.1.3 transfers data properly but
   the ftp server sends a message "426: Failure writing network stream".
A. This is a bug the vsftpd-1.1.3 and is promised to be fixed in a later
   release according to chris@scary.beasts.org. The bug is that after
   writing the file out using sendfile(), the code checks the errno value
   irrespective of whether an error had occurred.

Q. I cannot use the "active" mode ftp with vsftpd. Whats up with that?
A. See limitations about how Alpine4Linux does not support sending file
   descriptors between processes.

Q. I cannot make the standard Linux "ftp" client use the Alpine stack. It
   simply hangs after connecting to the remote ftp server.
A. Yeah. It sucks. My speculation is that ftp is using getc/putc
   to read/write to the socket and this is going to glibc instead of
   Alpine4Linux. I have not spent any reasonable amount of time 
   diagnosing it so I could be completely wrong.
 

ActiveState Acquires Perl Cloud Company, Phenona, to Accelerate Cloud Strategy

ActiveState Acquires Perl Cloud Company, Phenona, to Accelerate Cloud Strategy

Vancouver, BC – June 14, 2011 – ActiveState, the dynamic language experts offering solutions for Perl, Python, Tcl and other web languages, is announcing today that it has acquired Phenona, a Perl Platform-as-a-Service (PaaS) cloud company based in Seattle, WA. This announcement follows on the heels of ActiveState’s Stackato cloud platform announcement last month, an enterprise cloud offering based on VMware’s Cloud Foundry. Phenona’s 15-year-old founder, Daniil Kulchenko, will join the ActiveState team part-time since he is still a student in high school.

Read more

Real-Time Text Over Jabber/XMPP/Google Talk

From Slashdot: Real-Time Text Over Jabber/XMPP/Google Talk

"Geeks who miss the UNIX 'talk' days, have a new modern savior: XMPP.org has published the new XEP-0301 Real-Time Text standard, which allows streaming text that is continuously transmitted as it is typed or otherwise composed. It allows conversational use of text, where people interactively converse with each other."

XMPP.org

Brain-inspired computing with nanoelectronic programmable synapses

Good article:
Brain-inspired computing with nanoelectronic programmable synapses 

I have done a little bit of research in to Non von Neumann Architectures.
Our current way of doing computers works great for what it is, but it's limited.


A design like the one in this article can tackle some problems that would  be difficult for traditional CPU's.

SVLUG July 6th meeting: Wired Roots: Applications of Wireless Sensor Networks for Habitat Monitoring in the Wilds of Mt. Hamilton

It was a good talk, interesting work on remote sensing and low power networking. 


Here are some links from the talk. 

http://blueoakranchreserve.org/

http://ekoview.blueoakranchreserve.org/

http://en.wikipedia.org/wiki/Blue_Oak_Ranch_Reserve

---------- Forwarded message ----------
From: Rick Moen
Date: Tue, 5 Jul 2011 10:43:13 -0700
Subject: [svlug] [svlug-announce] SVLUG July 6th meeting: Wired Roots:
Applications    of Wireless     Sensor Networks for Habitat Monitoring in
the     Wilds of Mt. Hamilton

WHEN:

 Wednesday, July 6th, 2011
 7pm-9pm

MAIN PRESENTATION

 TOPIC:
   Wired Roots:  Applications of Wireless Sensor Networks for Habitat
   Monitoring in the Wilds of Mt. Hamilton

 PRESENTED BY:
   Michael Hamilton,
   Dir. of U.C. Berkeley's Blue Oak Ranch Reserve

 TOPIC SUMMARY:
   Dr. Hamilton will discuss a brief history of the Blue Oak Ranch
   Reserve, a 5,000 acre ecological reserve perched upon Mt Hamilton
   below the Lick Observatory -- and how technologies such as Ubuntu
   Linux, wireless networks, sensors, and imagers will transform our
   understanding of ecological processes and interactions of species
   and the environment.  He will conclude with a brief discussion of
   the San Jose Climate Clock project.

 ABOUT THE PRESENTER:
    Michael Hamilton is an ecologist, conservation biologist, and the
    reserve director of the Blue Oak Ranch Reserve, a unit of the UC
    Natural Reserve System, located near Mount Hamilton in the Diablo
    Range, due east of San Jose, California. He received his Ph.D. from
    Cornell University in 1983, and holds B.S. and M.S. degrees in
    biology and ecology from California State Polytechnic University,
    Pomona.

    Michael is one of several founders, a past board member, and
    currently serves as scientific advisor for the Society for
    Conservation GIS, an organization with 3000 members worldwide that
    trains and supports academic and NGO professionals involved in
    nature protection and conservation biology using remote sensing and
    Geographic Information Systems (GIS) technologies. His professional
    activities have taken him throughout the United States and Mexico,
    and to many continents and countries including Europe, Africa,
    Australia, South America, and French Polynesia.

LOCATION:

 Symantec
 VCAFE Facility
 350 Ellis Street (near E. Middlefield Road)
 Mountain View, CA 94043

 Directions on how to get there are listed at:

   http://www.svlug.org/directions/veritas.php

 We've tried our very best for these directions to be accurate.
 If you have any improvements to make, please let SVLUG's volunteers know!
 webmaster at svlug.org

POST-MEETING GATHERING:

 If you just can't get enough, a smaller group usually goes to a local
 restaurant/diner after the meeting:  Frankie, Johnnie & Luigi, Too,
 939 West El Camino Real between Shoreline and Castro, Mountain View.

We look forward to seeing you there!

_______________________________________________
svlug-announce mailing list
http://lists.svlug.org/lists/listinfo/svlug-announce

_______________________________________________
svlug mailing list
http://lists.svlug.org/lists/listinfo/svlug

PerlMonks - The Monastery Gates

http://www.perlmonks.org/

WebOS

http://en.m.wikipedia.org/wiki/WebOS

Ozma: Extending Scala With Oz Concurrency

I want to go through this to understand it better.  The "Unusual Syntax" sounds like a major hurdle and most of these feartures probably could be added to more traditional languages.  C++, JAVA, RUBY, PERL, etc.



From Slashdot: Announcing Ozma: Extending Scala With Oz Concurrency

"Programming for concurrency makes sense in developing for both large scales (cloud computing) and small (multicore CPUs). Some languages were designed for concurrency and distribution; One of those languages is Oz, which provides advanced primitives regarding concurrency and distribution. Oz is mostly declarative, a paradigm that encompasses functional and logic programming. Despite its innovative features and expressiveness, Oz never made it into the wide developer community; one reason is its unusual syntax." Read on to learn about an effort to bring Oz's concurrency features to more programmers. setori88 continues: "But first, some background: Martin Odersky, in designing the Scala programming language, recognized the need for community acceptance of the kind that Oz lacked. He designed Scala as being both close to Java (in terms of syntax and concepts) and interoperable with existing Java libraries. Today, Scala seems to be the best hope for making functional programming accessible to programmers.

Although Scala has no language feature related to concurrency, the advanced library Akka, inspired by Erlang, provides Scala programmers with concurrent and distributed concepts.

Now comes a project attempting to popularize the concurrency concepts of Oz, called Ozma. Ozma implements the full Scala specification and runs on the Mozart VM. It extends Scala with dataflow variables, declarative concurrency, lazy declarative concurrency, and message-passing concurrency based on ports. Ozma extends the duality of Scala, namely the combination of functional and object styles, to concurrent programming."

The Mozart Programming System
http://www.mozart-oz.org/ The Mozart Programming System is an advanced development platform for intelligent, distributed applications. The system is the result of a decade of research in programming language design and implementation, constraint-based inference, distributed computing, and human-computer interfaces.

Mozart is based on the Oz language, which supports declarative programming, object-oriented programming, constraint programming, and concurrency as part of a coherent whole. For distribution, Mozart provides a true network transparent implementation with support for network awareness, openness, and fault tolerance. Mozart supports multi-core programming with its network transparent distribution and is an ideal platform for both general-purpose distributed applications as well as for hard problems requiring sophisticated optimization and inferencing abilities. We have developed many applications including sophisticated collaborative tools, multi-agent systems, and digital assistants, as well as applications in natural language understanding and knowledge representation, in scheduling and time-tabling, and in placement and configuration.

MemCacheD

http://en.wikipedia.org/wiki/Memcached

Manage Open Source Software | Black Duck Software

http://www.blackducksoftware.com/

Li Mei’s list of debugging secrets

http://www.eetimes.com/design/embedded/4217189/Li-Mei-s-list-of-debugging-secrets?cid=NL_Embedded&Ecosystem=embedded

BayLISA had a some great talks last week

BayLISA a group dedicated to Sysadmins, had a some great talks last week.

• Force 10 bare metal provisioning demo.  (IaaS, Cloud component)
• Arista Networks vmtracer demo
• Logstash, with Jordan Sissel 
• FreeNAS BSD based Open Source NAS sponsored by IX Systems.

Logstash really looks a lot like a Open source version of Splunk.

What is the Cloud?

Components of a Cloud.
There are three main categories of cloud computing service, called the SPI model (SaaS, PaaS, IaaS).

Anything as a Serivce (XaaS) is a term used to all of these.

1. Software as a Service (SaaS) is a software distribution model in which applications are hosted by a vendor or service provider and made available to customers over a network, typically the Internet.
2. Platform as a Service (PaaS) is a paradigm for delivering operating systems and associated services over the Internet without downloads or installation.
3. Infrastructure as a Service (IaaS) involves outsourcing the equipment used to support operations, including storage, hardware, servers and networking components.

What is Infrastructure as a Service (IaaS)?
Infrastructure as a Service is a provision model in which an organization outsources the equipment used to support operations, including storage, hardware, servers and networking components. The service provider owns the equipment and is responsible for housing, running and maintaining it. The client typically pays on a per-use basis.

Characteristics and components of IaaS include:

• Utility computing service and billing model.
• Automation of administrative tasks.
• Dynamic scaling.
• Desktop virtualization.
• Policy-based services.
• Internet connectivity.
• Virtual Servers, and Virtualization

Sometimes referred to as (HaaS) Hardware as a Service.


Platform as a Service (PaaS) is a way to rent hardware, operating systems, storage and network capacity over the Internet. The service delivery model allows the customer to rent virtualized servers and associated services for running existing applications or developing and testing new ones.

Platform as a Service (PaaS) is an outgrowth of Software as a Service (SaaS), a software distribution model in which hosted software applications are made available to customers over the Internet. PaaS has several advantages for developers. With PaaS, operating system features can be changed and upgraded frequently. Geographically distributed development teams can work together on software development projects. Services can be obtained from diverse sources that cross international boundaries. Initial and ongoing costs can be reduced by the use of infrastructure services from a single vendor rather than maintaining multiple hardware facilities that often perform duplicate functions or suffer from incompatibility problems. Overall expenses can also be minimized by unification of programming development efforts.
On the downside, PaaS involves some risk of "lock-in" if offerings require proprietary service interfaces or development languages. Another potential pitfall is that the flexibility of offerings may not meet the needs of some users whose requirements rapidly evolve.
 

Software as a Service (SaaS) is a software distribution model in which applications are hosted by a vendor or service provider and made available to customers over a network, typically the Internet.

SaaS is becoming an increasingly prevalent delivery model as underlying technologies that support Web services and service-oriented architecture (SOA) mature and new developmental approaches, such as Ajax, become popular. Meanwhile, broadband service has become increasingly available to support user access from more areas around the world.
SaaS is closely related to the ASP (application service provider) and on demand computing software delivery models. IDC identifies two slightly different delivery models for SaaS. The hosted application management (hosted AM) model is similar to ASP: a provider hosts commercially available software for customers and delivers it over the Web. In the software on demand model, the provider gives customers network-based access to a single copy of an application created specifically for SaaS distribution.
Benefits of the SaaS model include:

• easier administration
• automatic updates and patch management
• compatibility: All users will have the same version of software.
• easier collaboration, for the same reason
• global accessibility.

The traditional model of software distribution, in which software is purchased for and installed on personal computers, is sometimes referred to as software as a product.


SaaS can be build on PaaS on IaaS


-------------------------------
This is still a draft article from here on down.
I am in the process of still writting.
John 6/17/2011
-------------------------------

Examples:

• Amazon web services, BeanStalk, Elastic Compute EC2
• VMware
• Windows Azure
• SalesForce.com
• Google Gmail
• CloudBees

Scale out vs. Scale up.

Tools:
Enterprise Java three tier application
MySQL, MongoDB
NoSQL, CouchDB

Memory or Cache (Redis, memcached, Gemfire)

RabbitMQ, AMQP, JMS for messaging in the cloud. 
Amazon SNS, SQS

POJO? Apache Hadoop?  HDFS, HBase, Hive

Makara, an open-source software-project for cloud deployment and management managed by Red Hat

MAP/Reduce

CAP Theorem
C: Consistency: All nodes see the same data
A: Availablity: Node failure does not prevent survivors from operating
P: Partition Tolerance: System works even though message are lost

http://en.wikipedia.org/wiki/Cloud_computing
http://en.wikipedia.org/wiki/List_of_cloud_computing_providers
http://en.wikipedia.org/wiki/Cloud_%28operating_system%29

Redis key-value store server.

Redis is an advanced key-value store.

It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.

Written in ANSI C, it's open-source, networked, in-memory, persistent and  journaled. 

Supported languages or language bindings include C, C++, C#, Clojure, Common Lisp, Erlang, Haskell, Java, JavaScript, Lua, Objective-C, Perl, PHP, Python, R, Ruby, Scala, Go, and Tcl.
As of 15 March 2010, development of Redis is sponsored by VMware.

http://redis.io/
http://en.wikipedia.org/wiki/Redis (data store)
Redis: Lightweight key/value Store That Goes the Extra Mile
Evolving the Key/Value Programming Model to a Higher Level 

Advanced Message Queuing Protocol

The Advanced Message Queuing Protocol (AMQP) is an open standard application layer protocol for message-oriented middleware (MOM).  The defining features of AMQP are message orientation, queuing, routing (including point-to-point and publish-and-subscribe), reliability and security.


It's similar to JMS for Java but not language specific. 

http://www.amqp.org/confluence/display/AMQP/Advanced+Message+Queuing+Protocol

Linux based PID Servomotor Control

Great article that talk about implemented the PID, or the Proportional Integral Derivative, algorithm to make a 2 wheeled robot that balences like a segway. 

Linux based PID Control

How Skype gets round firewalls

UPDATE: This protocol is now incorporated in to STUN , TURN and ICE.     Global IP Solutions, the company that made the VOIP system that Skype was using was bought by Google shortly after Skype developed their own technology.  Google then made the original skype technology freely available as WebRTC now supported by many HTML5 browsers.  Microsoft then bought Skype. ----- REPOST FROM: http://www.heise-security.co.uk/articles/82481  No longer on the web. I want to point out this is the same method described in my Paper: Method of passing bi-directional data between two firewalls. 12 /15/2006

 

Jürgen Schmidt

The hole trick

How Skype & Co. get round firewalls

Peer-to-peer software applications are a network administrator's nightmare. In order to be able to exchange packets with their counterpart as directly as possible they use subtle tricks to punch holes in firewalls, which shouldn't actually be letting in packets from the outside world.
Increasingly, computers are positioned behind firewalls to protect systems from internet threats. Ideally, the firewall function will be performed by a router, which also translates the PC's local network address to the public IP address (Network Address Translation, or NAT). This means an attacker cannot directly address the PC from the outside - connections have to be established from the inside.
This is of course a problem when two computers behind NAT firewalls require to talk directly to each other - if, for example, their users want to call each other using Voice over IP (VoIP). The dilemma is clear - whichever party calls the other, the recipient's firewall will decline the apparent attack and will simply discard the data packets. The telephone call doesn't happen. Or at least that's what a network administrator would expect.

Punched

But anyone who has used the popular internet telephony software Skype knows that it works as smoothly behind a NAT firewall as it does if the PC is connected directly to the internet. The reason for this is that the inventors of Skype and similar software have come up with a solution.
Naturally every firewall must also let packets through into the local network - after all the user wants to view websites, read e-mails, etc. The firewall must therefore forward the relevant data packets from outside, to the workstation computer on the LAN. However it only does so, when it is convinced that a packet represents the response to an outgoing data packet. A NAT router therefore keeps tables of which internal computer has communicated with which external computer and which ports the two have used.
The trick used by VoIP software consists of persuading the firewall that a connection has been established, to which it should allocate subsequent incoming data packets. The fact that audio data for VoIP is sent using the connectionless UDP protocol acts to Skype's advantage. In contrast to TCP, which includes additional connection information in each packet, with UDP, a firewall sees only the addresses and ports of the source and destination systems. If, for an incoming UDP packet, these match an NAT table entry, it will pass the packet on to an internal computer with a clear conscience.

Switching

The switching server, with which both ends of a call are in constant contact, plays an important role when establishing a connection using Skype. This occurs via a TCP connection, which the clients themselves establish. The Skype server therefore always knows under what address a Skype user is currently available on the internet. Where possible the actual telephone connections do not run via the Skype server; rather, the clients exchange data directly.
Let's assume that Alice wants to call her friend Bob. Her Skype client tells the Skype server that she wants to do so. The Skype server already knows a bit about Alice. From the incoming query it sees that Alice is currently registered at the IP address 1.1.1.1 and a quick test reveals that her audio data always comes from UDP port 1414. The Skype server passes this information on to Bob's Skype client, which, according to its database, is currently registered at the IP address 2.2.2.2 and which, by preference uses UDP port 2828.

Step 1: Alice tries to call Bob, which signals Skype.

Bob's Skype program then punches a hole in its own network firewall: It sends a UDP packet to 1.1.1.1 port 1414. This is discarded by Alice's firewall, but Bob's firewall doesn't know that. It now thinks that anything which comes from 1.1.1.1 port 1414 and is addressed to Bob's IP address 2.2.2.2 and port 2828 is legitimate - it must be the response to the query which has just been sent.

Step 2: Bob tries to reach Alice, which punches a hole through Bob's Firewall.

Now the Skype server passes Bob's coordinates on to Alice, whose Skype application attempts to contact Bob at 2.2.2.2:2828. Bob's firewall sees the recognized sender address and passes the apparent response on to Bob's PC - and his Skype phone rings.

Step 3: Alice finally reaches Bobs computer through the hole.

Doing the rounds

This description is of course somewhat simplified - the details depend on the specific properties of the firewalls used. But it corresponds in principle to our observations of the process of establishing a connection between two Skype clients, each of which was behind a Linux firewall. The firewalls were configured with NAT for a LAN and permitted outgoing UDP traffic.
Linux' NAT functions have the VoIP friendly property of, at least initially, not changing the ports of outgoing packets. The NAT router merely replaces the private, local IP address with its own address - the UDP source port selected by Skype is retained. Only when multiple clients on the local network use the same source port does the NAT router stick its oar in and reset the port to a previously unused value. This is because each set of two IP addresses and ports must be able to be unambiguously assigned to a connection between two computers at all times. The router will subsequently have to reconstruct the internal IP address of the original sender from the response packet's destination port.
Other NAT routers will try to assign ports in a specific range, for example ports from 30,000 onwards, and translate UDP port 1414, if possible, to 31414. This is, of course, no problem for Skype - the procedure described above continues to work in a similar manner without limitations.
It becomes a little more complicated if a firewall simply assigns ports in sequence, like Check Point's FireWall-1: the first connection is assigned 30001, the next 30002, etc. The Skype server knows that Bob is talking to it from port 31234, but the connection to Alice will run via a different port. But even here Skype is able to outwit the firewall. It simply runs through the ports above 31234 in sequence, hoping at some point to stumble on the right one. But if this doesn't work first go, Skype doesn't give up. Bob's Skype opens a new connection to the Skype server, the source port of which is then used for a further sequence of probes.

Skype can do port scans. Here it succeeds on port 38901 and connects through the firewall.

Nevertheless, in very active networks Alice may not find the correct, open port. The same also applies for a particular type of firewall, which assigns every new connection to a random source port. The Skype server is then unable to tell Alice where to look for a suitable hole in Bob's firewall.
However, even then, Skype doesn't give up. In such cases a Skype server is then used as a relay. It accepts incoming connections from both Alice and Bob and relays the packets onwards. This solution is always possible, as long as the firewall permits outgoing UDP traffic. It involves, however, an additional load on the infrastructure, because all audio data has to run through Skype's servers. The extended packet transmission times can also result in an unpleasant delay.
Use of the procedure described above is not limited to Skype and is known as "UDP hole punching". Other network services such as the Hamachi gaming VPN application, which relies on peer-to-peer communication between computers behind firewalls, use similar procedures. A more developed form has even made it to the rank of a standard - RFC 3489 "Simple Traversal of UDP through NAT" (STUN) describes a protocol which with two STUN clients can get around the restrictions of NAT with the help of a STUN server in many cases. The draft Traversal Using Relay NAT (TURN) protocol describes a possible standard for relay servers.

DIY hole punching

With a few small utilities, you can try out UDP hole punching for yourself. The tools required, hping2 and netcat, can be found in most Linux distributions. Local is a computer behind a Linux firewall (local-fw) with a stateful firewall which only permits outgoing (UDP) connections. For simplicity, in our test the test computer remote was connected directly to the internet with no firewall.
Firstly start a UDP listener on UDP port 14141 on the local/1 console behind the firewall:

local/1# nc -u -l -p 14141

An external computer "remote" then attempts to contact it.

remote# echo "hello" | nc -p 53 -u local-fw 14141

However, as expected nothing is received on local/1 and, thanks to the firewall, nothing is returned to remote. Now on a second console, local/2, hping2, our universal tool for generating IP packets, punches a hole in the firewall:

local/2# hping2 -c 1 -2 -s 14141 -p 53 remote

As long as remote is behaving itself, it will send back a "port unreachable" response via ICMP - however this is of no consequence. On the second attempt

remote# echo "hello" | nc -p 53 -u local-fw 14141

the netcat listener on console local/1 then coughs up a "hello" - the UDP packet from outside has passed through the firewall and arrived at the computer behind it.
Network administrators who do not appreciate this sort of hole in their firewall and are worried about abuse, are left with only one option - they have to block outgoing UDP traffic, or limit it to essential individual cases. UDP is not required for normal internet communication anyway - the web, e-mail and suchlike all use TCP. Streaming protocols may, however, encounter problems, as they often use UDP because of the reduced overhead.
Astonishingly, hole punching also works with TCP. After an outgoing SYN packet the firewall / NAT router will forward incoming packets with suitable IP addresses and ports to the LAN even if they fail to confirm, or confirm the wrong sequence number (ACK). Linux firewalls at least, clearly fail to evaluate this information consistently. Establishing a TCP connection in this way is, however, not quite so simple, because Alice does not have the sequence number sent in Bob's first packet. The packet containing this information was discarded by her firewall.

WebRTC - bringing real time communications to the web.

WebRTC - an open real-time communications project

WebRTC offers web application developers the ability to write rich, realtime multimedia applications (think video chat) on the web, without requiring plugins, downloads or installs. It's purpose is to help build a strong RTC platform that works across multiple web browsers, across multiple platforms.


https://sites.google.com/site/webrtc/blog

Low Level Virtual Machine (LLVM)

LLVM is a really interesting alternative to java and it's bytecode jvm.
What's interesting is it can be run in the web browsers Javascript enterpreter allow this to run on web site.

So you can compile C & C++ code to run on the web in a javascript VM.

> Deepak  - Apple is funding the project.llvm-gcc is the default compiler which is shipped with Mac OS and IOS SDK.


http://llvm.org/

Low Level Virtual Machine (LLVM) is a compiler infrastructure, written in C++, which is designed for compile-time, link-time, run-time, and "idle-time" optimization of programs written in arbitrary programming languages. Originally implemented for C/C++, the language-independent design (and the success) of LLVM has since spawned a wide variety of front ends, including Objective-C, Fortran, Ada, Haskell, Java bytecode, Python, Ruby, ActionScript, GLSL, and others.

Emscripten is an LLVM-to-JavaScript compiler. It takes LLVM bitcode (which can be generated from C/C++, using llvm-gcc or clang, or any other language that can be converted into LLVM) and compiles that into JavaScript, which can be run on the web (or anywhere else JavaScript can run).

From Slashdot: Doom Ported To the Web

"Ever since Id Software released the Doom source code under the GPL, it's been ported to platform after platform. Now, you can play Doom compiled to JavaScript on the web, using standard web technologies like Canvas and without any plugins. If your browser has trouble running it, here's a screencast." The translation was accomplished using Emscripten, a Javascript backend for LLVM. As per the GPL, full source code is available. Pretty neat.

Use the PS3 for your project?

http://www.freebsd.org/news/status/report-2010-07-2010-09.html#FreeBSD-on-the-Playstation-3

Apple TV hacked and turned in to web server.

An Apple TV-Based Webserver - Slashdot

Lua and embedded programming.

Designing a Programming Language For Embeddability - Slashdot

Browse the web from the US

I friend of my is in the Middle east and wishes to browse the web from the US.

I am going to try to setup open VPN and tinyproxy on my Virtual Server Co-Lo for him and see how that works.

https://banu.com/tinyproxy/

http://openvpn.net/index.php/open-source.html

Bash: Find the Max in a list of Integers

Max is a little shells script that will find the maximum value and returns that value and the line it was found at.

max

#!/bin/bash
MAX=0
ELE=0
while read line; do
ELE=$[ $ELE + 1 ]
if [ "$line" -gt "$MAX" ]; then
MAX=$line
POS=$ELE
fi
done
echo "$MAX $POS"


Usage Example

-bash-3.2$ cat ttt
1000000001
1000000000002
1000000000007
1000000000033
9999
1000000000000
1000000000005
-bash-3.2$ ./max < ttt
1000000000033 4
-bash-3.2$


Hackers Find New Way To Cheat On Wall Street

From Slashdot:

"The high-speed trading exchanges that conduct the business of buying and selling stocks and mutual funds are so fast that hackers can introduce delays of a few microseconds completely unnoticed by today's network monitoring technology — and manipulate prices in the process to reap millions of dollars to the detriment of everyone else, InfoWorld's Bill Snyder reports. This kind of activity creates new reason to distrust Wall Street and shows how the computer networks we all rely on for conducting business and moving information are ripe for undetectable hacking."

Preserving Great Tech For Posterity — the 6502

From Slashdot:

"For great old hardware products like the MOS 6502 (used in the Apple II, the C64, the Nintendo NES), the details of the designs have been lost or forgotten. While there have been great efforts to reverse engineer the 6502 from the outside, there has not been the hardware equivalent of the source code — until now. As Russell Cox states: 'A team of three people accumulated a bunch of 6502 chips, applied sulfuric acid to them to strip the casing and expose the actual chips, used a high-resolution photomicroscope to scan the chips, applied computer graphics techniques to build a vector representation of the chip, and finally derived from the vector form what amounts to the circuit diagram of the chip: a list of all 3,510 transistors with inputs, outputs, and what they're connected to. Combining that with a fairly generic (and, as these things go, trivial) "transistor circuit" simulator written in JavaScript and some HTML5 goodness, they created an animated 6502 web page that lets you watch the voltages race around the chip as it executes. For more, see their web site visual6502.org.'"

Update All Scripts in One Swoop

First this is old hat for most admins, so they will sigh a collective yawn.

Now, let's say, like me you have a couple of dozen scripts written in wish8.4 and you want to update all the scripts at once to wish8.5.

sed -i 's/wish8.4/wish8.5' *

• sed - the classic stream editor in Unix (yes even linux and windows have it)
  
• -i - option, edit "in-place", meaning don't output the result, just change silently
  
• s /x/y/ - substitute y for x
• * - command line glob

NOTE the last was for LINUX on BSD use -i .bak 's/....
(added 2011-01-30)

tc

Putin Orders Russian Move To GNU/Linux

From Slashdot:

"Vladimir Putin has signed an order calling for Russian federal authorities to move to GNU/Linux, and for the creation of 'a single repository of free software used in the federal bodies of executive power.' There have been a number of Russian projects to roll out free software, notably in the educational sector, but none so far has really taken off. With the backing of Putin, could this be the breakthrough free software has been waiting for?"

Here is another link to the story:
http://mashable.com/2010/12/27/vladimir-putin-free-software-by-2015/

BSD Philosophy

BSD Unix started at Berkeley University shortly after a very turbulent and rebellious time at Berkeley in the late 60's.
During that time there were several major movements happening all at once. The Free Speech, Anti-War, Hippies, Free City, think for yourself, Acid test, civil rights, and a tornado of other forces that lead to several all out riots on campus.
Other influences at Berkeley are a large Asian Population and the study of eastern cultures (Chinese, Japanese and India) and Philosophy.

Many of these memes later incorporated themselves into the BSD Unix culture and Philosophy.

Respect for elders, New isn't always better


Unix and BSD Unix are old operating systems by our modern perception. But, if you were to think this was a bad thing you'd be totally wrong.


At the root of most Modern Operating Systems are design concepts first pioneered in Unix and BSD. Many way DOS, Windows NT,2000,XP, Mac OS X, NeXT OS, operate are based on UNIX concepts.

Change isn't the same as improvement

"The Great Wall of China wasn't built by swapping any bricks in and out" - John Sokol.

"The devil you know is better than the one you don't"

Unix development started more than three decades ago, and the versions of operating system available today is the product of thousands upon thousands of developers hard work and constant improvements. Along the way, Unix has pioneered some very powerful design concepts and some elgant solutions to hard problems common in computing.

"If I had more time, I would have written a shorter letter." - Mark Twain


"I have made this letter longer than usual, because I lack the time to make it short" (Je n'ai fait celle-ci plus longue parceque je n'ai pas eu le loisir de la faire plus courte) - Blaise Pascal, Lettres Provinciales (1656-1657), no. 16.

"Anyone can make something complex, but it takes a genius to make it simple" - Albert Einstein

Elimination of unnecessary complexity is a Key part of the BSD Philosophy

Much of the "Unix way" is based on the idea of abstracting interfaces into the simplest possible terms, BSD trys to take this to it's limits.

Initially, this was simply because the OS was considered experimental, and the simplest possible interface, one that wouldn't change later, .was the most easiest to construct.
This spend time upfront to think of how to "do it the easy way" methodology has evolved into a many of the most powerful design concepts in OS design. Abstraction that started in Unix is the core of many parts of a modern OS. From Files, Devices, Sockets, Modules, Processes, Memory Management, File Systems, directories, and Pipes.

Even the Internet is based on BSD technology and concepts. The BSD 4.2 OS was the first to Implement TCP/IP and part of that was the concept of network Sockets and Ports. Many other more complex network technology had been around for a while, but BSD Sockets allow for two computers to communicate in a way that uses the exact same commands as file access. Open, Read, Write, Close. It also uses another concept, Streams.


Simplicity and elegance though consistent software interfaces

Where:

elegant : adj. [common; from mathematical usage] Combining simplicity, power, and a certain ineffable grace of design. Higher praise than `clever', `winning', or even cuspy.

"The challenge is keeping it simple while providing the people with functions. This is harder to do. Anyone can make something complex." Dauphin A German Chair company

Try to understand the intent of the original authors' style and intent, and make changes consistent with this


This means not just going in slashing and hacking changes into the Kernel or User space code without researching and thinking them through first. Find a clean way to do what ever your trying to do. Spaghetti code is not tolerated.


All too often a quick and dirty little hack that was meant to be temporary become permanent and hard to fix later.

KISS, Keep It Simple, Stupid or maybe Keep it Short and Simple, I like the first one.

Working on BSD code is basically working on other people code for the most part. Most likely what ever you do, someone else will have to work on after you.

Working on someone else code is something many application programmers realy suck at. Also they think using the latest cool tricks that just came out in the latest language spec or the new compiler supports it so it's really Cool.
F*ck them! These guys need to be slapped up side the head a few times.
I see time and time again the Linux guys keep making a mess for those who need to update there code and drivers over to the latest OS version. Often is is almost as bad as a cross platform port, nothing works out the box everything is broken and it becomes a major undertaking!

The Mac OS-X darwin they are mixing C++ and Objective C into the kernel source tree! It's full of broken locks and is nearly impossible to understand anymore. Leaving some messy and hard kernel hacking to the customers having to make there code work on a pig of an OS. I hope that doesn't sound like I'm venting too much, does it?


"A program with a colorful GUI is like a woman with too much makeup. Shiny on the outside - awful on the inside."

Quote from the NET



Are you going to run a gui or from the command line etc... over SSH?
If you want a gui I would go Linux maybe Fedora, for SSH use FreeBSD...I used to be against it but I am now a convert to the Church of BSD.

Why would I ever want a GUI? It may sound odd but I have never ran X on any of the 100's of FreeBSD systems I have worked on. Command line Interfaces Rull, and for most of the stuff I do is much faster. Even this web page is edited in "VI".


We are not Linux, We were here before them; and well be here after them

So what's the difference, much of the same code get's passed between BSD and Linux Right? That's only partly true. The Licenses are incompatable between the two for starters.

"I must say the linux community is a lot nicer than the unix
community. a negative comment on unix would warrant death
threats. With linux, it is like stirring up a nest of butterflies."
             -- Ken Thompson author of C Language. 1999 

BSD vs. Linux, Any questions?

FreeBSD, OpenBSD and SuSE 6.2 Eval Review by Keith Rankin:
"The differences between FreeBSD and Linux used to be much more obvious than they are now.
Now it comes down to theology. The BSD world is still the 'high church' or Druid Unix. Blood will be spilled on a stone altar at midnite when star systems are in a certain alignment to learn the ways of this tribe. Linux is a happier world. The spirits of Captain Kirk, Peter Pan and good beer come to mind."

Hackers and the bushido code

For much of the early life of BSD Unix was spent running campus MiniComputers. PDP/11, Vax 11/750 and the like.

One way to attain elite status (enlightenment or become the master or Guru) was to "break root", break in or even just be able to crash the server.

This was not viewed as a federal offense, or even as something inappropriate but as an exciting challenge and testing the limits of the box.

For the many years it was an arms race between the OS developers and students to see if the box could be crashed by malicoius code or commands, the end result is something rock solid.

Now crashing a windows box usualy requires little more then power it up, but most BSD systems can go for years without rebooting, crashing or having it security compromised. Even on my own servers, like the one that ran the LeonardoDicaprio.com site, despite countless attempts to take out the box, it had little effect.

Part of Hacker Philosophy directly stems from these early BSD days, although not all of it. One of the thing was to not damage or sabotage, but to report and inform people of the bug or security hole. Maybe a little Kilroy was here on the administrators console ;)

Reboots, re-compiles and Re/Installations are EVIL and to be avoided at all costs.


One important thing about BSD is the belief that you should be able to stress the system in any number of ways, like open up and infinite number of files, sockets, programs, memory, or any other resource and have the system respond gracefully. But this I mean not to have a Blue Screen of Death, or Kernel Panic.


Linux groups have installfests, while BSD groups have installathons. Clearly an installfests sounds festive and much happier then an installathons that is more like a Marathon to try to race through it.

We are the one true Path, all non-believers (can go to hell) are damned to hell

This is in the tradition of all great religions.

From: Nick Moffitt nick@zork.net , With some paraphrasing.


Tue, 22 Aug 2000 14:02:47
Basically, user groups nowadays are either:

 Lame talking-heads lecture series.

 Cowtow to the windows users installathons.


  I say that's bullsh*t, and we all know it. F*CK the newbies.

We'll hold installfests, sure, but we'll be perfectly justified in telling people to take their 386s with 15 year-old RLL drives and GO HOME. This will be a meeting of COOL PEOPLE to show off COOL STUFF!

We'll not be limited to a particular OS, either. We'll have Linux weenies, FreeBSD weenies, and cypherpunk weenies clutching their precious OpenBSD! We'll have the AMAZING GEORGE PERRY and his BEARDFUL OF FORTH!

I think that expresses the attitude fairly well...

BSD License

Do what is right and be hated by those around you. Do what is popular and be hated by history.
The BSD License allows anyone to use the code for any reason and any way with no restrictions other then credit is given to the Authors
Transparency Begets Trust - Expertise in niches, transparency in motives and thought process and owning up to mistakes publicly create a trust relationship" - Will Pate



In General people in the BSD world view people and Individuals who do not share code and improvements as pompous idiots. There are a ton of smart people in the BSD world and no one individual or company (M$) not sharing code is really going to make any difference, the reality is we probably wouldn't want there code anyhow.

Trust builds Confidence

They will loose the benefits of having 1000's of eyes screening the code for holes and bugs as well as tightening things up and making improvements.

This is very much same philosophy as the Private vs. Public research in almost any scientific field. If you keep your research to your self, then you don't get credited for it. Also your work never gets confirmed, there is a good chance of deluding yourself about how secure and bug free your code is. (take Microsoft for an example)

The BSD approach is consider by most people to be better for businesses, companies such as Apple and Microsoft benefited their users by incorporating BSD-licensed code without having to share there code or changes to this code.

When questioned, most GPL programmers did not want the work they released to be used in a manner they did not support.


The BSD philosophy seems to hold that creating and giving away code, then seeing it used by others, is victory and reward enough.


But most of the GPL supporters disapproved of allowing "others" to close off source code and hide enhancements.

Why the Church of BSD

BSD has been through many Holy Wars and has been called a religion by many people for a long time. Even it's experts are called Guru's , High Priests and Wizards. What BSD programmers do has sometimes been called VooDoo, Alchemy and Black Magic. People who violate our way of doing things are declared Heretics, and publicly flamed! We even have our own official canonical way of doing things and this is in the source code.

The Source code is our Bible.

It seems we are long over due for a formal Church of BSD.

Canonical is an adjective derived from canon. It essentially means "standard", "generally accepted" or "part of the back-story." Canonical in my context means reduced to the simplest and most significant form possible without loss of generality.
This word is usualy used by theologians and canon lawyers to refer to the canons of the Eastern Orthodox and Roman Catholic churches, adopted by ecumenical councils.

Hacker Slang - Holy Wars From answers.com
[from Usenet, but may predate it; common] n. flame wars over religious issues. The paper by Danny Cohen that popularized the terms big-endian and little-endian in connection with the LSB-first/MSB-first controversy was entitled On Holy Wars and a Plea for Peace. Great holy wars of the past have included ITS vs. Unix, Unix vs. VMS, BSD Unix vs. System V, C vs. Pascal, C vs. FORTRAN, etc. In the year 2003, popular favorites of the day are KDE vs. GNOME, vim vs. elvis, Linux vs. [Free|Net|Open]BSD. Hardy perennials include EMACS vs. VI, my personal computer vs. everyone else's personal computer, ad nauseam. The characteristic that distinguishes holy wars from normal technical disputes is that in a holy war most of the participants spend their time trying to pass off personal value choices and cultural attachments as objective technical evaluations. This happens precisely because in a true holy war, the actual substantive differences between the sides are relatively minor. See also theology.