What is the Cloud?

Components of a Cloud.
There are three main categories of cloud computing service, called the SPI model (SaaS, PaaS, IaaS).

Anything as a Serivce (XaaS) is a term used to all of these.

1. Software as a Service (SaaS) is a software distribution model in which applications are hosted by a vendor or service provider and made available to customers over a network, typically the Internet.
2. Platform as a Service (PaaS) is a paradigm for delivering operating systems and associated services over the Internet without downloads or installation.
3. Infrastructure as a Service (IaaS) involves outsourcing the equipment used to support operations, including storage, hardware, servers and networking components.

What is Infrastructure as a Service (IaaS)?
Infrastructure as a Service is a provision model in which an organization outsources the equipment used to support operations, including storage, hardware, servers and networking components. The service provider owns the equipment and is responsible for housing, running and maintaining it. The client typically pays on a per-use basis.

Characteristics and components of IaaS include:

• Utility computing service and billing model.
• Automation of administrative tasks.
• Dynamic scaling.
• Desktop virtualization.
• Policy-based services.
• Internet connectivity.
• Virtual Servers, and Virtualization

Sometimes referred to as (HaaS) Hardware as a Service.


Platform as a Service (PaaS) is a way to rent hardware, operating systems, storage and network capacity over the Internet. The service delivery model allows the customer to rent virtualized servers and associated services for running existing applications or developing and testing new ones.

Platform as a Service (PaaS) is an outgrowth of Software as a Service (SaaS), a software distribution model in which hosted software applications are made available to customers over the Internet. PaaS has several advantages for developers. With PaaS, operating system features can be changed and upgraded frequently. Geographically distributed development teams can work together on software development projects. Services can be obtained from diverse sources that cross international boundaries. Initial and ongoing costs can be reduced by the use of infrastructure services from a single vendor rather than maintaining multiple hardware facilities that often perform duplicate functions or suffer from incompatibility problems. Overall expenses can also be minimized by unification of programming development efforts.
On the downside, PaaS involves some risk of "lock-in" if offerings require proprietary service interfaces or development languages. Another potential pitfall is that the flexibility of offerings may not meet the needs of some users whose requirements rapidly evolve.
 

Software as a Service (SaaS) is a software distribution model in which applications are hosted by a vendor or service provider and made available to customers over a network, typically the Internet.

SaaS is becoming an increasingly prevalent delivery model as underlying technologies that support Web services and service-oriented architecture (SOA) mature and new developmental approaches, such as Ajax, become popular. Meanwhile, broadband service has become increasingly available to support user access from more areas around the world.
SaaS is closely related to the ASP (application service provider) and on demand computing software delivery models. IDC identifies two slightly different delivery models for SaaS. The hosted application management (hosted AM) model is similar to ASP: a provider hosts commercially available software for customers and delivers it over the Web. In the software on demand model, the provider gives customers network-based access to a single copy of an application created specifically for SaaS distribution.
Benefits of the SaaS model include:

• easier administration
• automatic updates and patch management
• compatibility: All users will have the same version of software.
• easier collaboration, for the same reason
• global accessibility.

The traditional model of software distribution, in which software is purchased for and installed on personal computers, is sometimes referred to as software as a product.


SaaS can be build on PaaS on IaaS


-------------------------------
This is still a draft article from here on down.
I am in the process of still writting.
John 6/17/2011
-------------------------------

Examples:

• Amazon web services, BeanStalk, Elastic Compute EC2
• VMware
• Windows Azure
• SalesForce.com
• Google Gmail
• CloudBees

Scale out vs. Scale up.

Tools:
Enterprise Java three tier application
MySQL, MongoDB
NoSQL, CouchDB

Memory or Cache (Redis, memcached, Gemfire)

RabbitMQ, AMQP, JMS for messaging in the cloud. 
Amazon SNS, SQS

POJO? Apache Hadoop?  HDFS, HBase, Hive

Makara, an open-source software-project for cloud deployment and management managed by Red Hat

MAP/Reduce

CAP Theorem
C: Consistency: All nodes see the same data
A: Availablity: Node failure does not prevent survivors from operating
P: Partition Tolerance: System works even though message are lost

http://en.wikipedia.org/wiki/Cloud_computing
http://en.wikipedia.org/wiki/List_of_cloud_computing_providers
http://en.wikipedia.org/wiki/Cloud_%28operating_system%29

Redis key-value store server.

Redis is an advanced key-value store.

It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.

Written in ANSI C, it's open-source, networked, in-memory, persistent and  journaled. 

Supported languages or language bindings include C, C++, C#, Clojure, Common Lisp, Erlang, Haskell, Java, JavaScript, Lua, Objective-C, Perl, PHP, Python, R, Ruby, Scala, Go, and Tcl.
As of 15 March 2010, development of Redis is sponsored by VMware.

http://redis.io/
http://en.wikipedia.org/wiki/Redis (data store)
Redis: Lightweight key/value Store That Goes the Extra Mile
Evolving the Key/Value Programming Model to a Higher Level 

Advanced Message Queuing Protocol

The Advanced Message Queuing Protocol (AMQP) is an open standard application layer protocol for message-oriented middleware (MOM).  The defining features of AMQP are message orientation, queuing, routing (including point-to-point and publish-and-subscribe), reliability and security.


It's similar to JMS for Java but not language specific. 

http://www.amqp.org/confluence/display/AMQP/Advanced+Message+Queuing+Protocol

Linux based PID Servomotor Control

Great article that talk about implemented the PID, or the Proportional Integral Derivative, algorithm to make a 2 wheeled robot that balences like a segway. 

Linux based PID Control

How Skype gets round firewalls

UPDATE: This protocol is now incorporated in to STUN , TURN and ICE.     Global IP Solutions, the company that made the VOIP system that Skype was using was bought by Google shortly after Skype developed their own technology.  Google then made the original skype technology freely available as WebRTC now supported by many HTML5 browsers.  Microsoft then bought Skype. ----- REPOST FROM: http://www.heise-security.co.uk/articles/82481  No longer on the web. I want to point out this is the same method described in my Paper: Method of passing bi-directional data between two firewalls. 12 /15/2006

 

Jürgen Schmidt

The hole trick

How Skype & Co. get round firewalls

Peer-to-peer software applications are a network administrator's nightmare. In order to be able to exchange packets with their counterpart as directly as possible they use subtle tricks to punch holes in firewalls, which shouldn't actually be letting in packets from the outside world.
Increasingly, computers are positioned behind firewalls to protect systems from internet threats. Ideally, the firewall function will be performed by a router, which also translates the PC's local network address to the public IP address (Network Address Translation, or NAT). This means an attacker cannot directly address the PC from the outside - connections have to be established from the inside.
This is of course a problem when two computers behind NAT firewalls require to talk directly to each other - if, for example, their users want to call each other using Voice over IP (VoIP). The dilemma is clear - whichever party calls the other, the recipient's firewall will decline the apparent attack and will simply discard the data packets. The telephone call doesn't happen. Or at least that's what a network administrator would expect.

Punched

But anyone who has used the popular internet telephony software Skype knows that it works as smoothly behind a NAT firewall as it does if the PC is connected directly to the internet. The reason for this is that the inventors of Skype and similar software have come up with a solution.
Naturally every firewall must also let packets through into the local network - after all the user wants to view websites, read e-mails, etc. The firewall must therefore forward the relevant data packets from outside, to the workstation computer on the LAN. However it only does so, when it is convinced that a packet represents the response to an outgoing data packet. A NAT router therefore keeps tables of which internal computer has communicated with which external computer and which ports the two have used.
The trick used by VoIP software consists of persuading the firewall that a connection has been established, to which it should allocate subsequent incoming data packets. The fact that audio data for VoIP is sent using the connectionless UDP protocol acts to Skype's advantage. In contrast to TCP, which includes additional connection information in each packet, with UDP, a firewall sees only the addresses and ports of the source and destination systems. If, for an incoming UDP packet, these match an NAT table entry, it will pass the packet on to an internal computer with a clear conscience.

Switching

The switching server, with which both ends of a call are in constant contact, plays an important role when establishing a connection using Skype. This occurs via a TCP connection, which the clients themselves establish. The Skype server therefore always knows under what address a Skype user is currently available on the internet. Where possible the actual telephone connections do not run via the Skype server; rather, the clients exchange data directly.
Let's assume that Alice wants to call her friend Bob. Her Skype client tells the Skype server that she wants to do so. The Skype server already knows a bit about Alice. From the incoming query it sees that Alice is currently registered at the IP address 1.1.1.1 and a quick test reveals that her audio data always comes from UDP port 1414. The Skype server passes this information on to Bob's Skype client, which, according to its database, is currently registered at the IP address 2.2.2.2 and which, by preference uses UDP port 2828.

Step 1: Alice tries to call Bob, which signals Skype.

Bob's Skype program then punches a hole in its own network firewall: It sends a UDP packet to 1.1.1.1 port 1414. This is discarded by Alice's firewall, but Bob's firewall doesn't know that. It now thinks that anything which comes from 1.1.1.1 port 1414 and is addressed to Bob's IP address 2.2.2.2 and port 2828 is legitimate - it must be the response to the query which has just been sent.

Step 2: Bob tries to reach Alice, which punches a hole through Bob's Firewall.

Now the Skype server passes Bob's coordinates on to Alice, whose Skype application attempts to contact Bob at 2.2.2.2:2828. Bob's firewall sees the recognized sender address and passes the apparent response on to Bob's PC - and his Skype phone rings.

Step 3: Alice finally reaches Bobs computer through the hole.

Doing the rounds

This description is of course somewhat simplified - the details depend on the specific properties of the firewalls used. But it corresponds in principle to our observations of the process of establishing a connection between two Skype clients, each of which was behind a Linux firewall. The firewalls were configured with NAT for a LAN and permitted outgoing UDP traffic.
Linux' NAT functions have the VoIP friendly property of, at least initially, not changing the ports of outgoing packets. The NAT router merely replaces the private, local IP address with its own address - the UDP source port selected by Skype is retained. Only when multiple clients on the local network use the same source port does the NAT router stick its oar in and reset the port to a previously unused value. This is because each set of two IP addresses and ports must be able to be unambiguously assigned to a connection between two computers at all times. The router will subsequently have to reconstruct the internal IP address of the original sender from the response packet's destination port.
Other NAT routers will try to assign ports in a specific range, for example ports from 30,000 onwards, and translate UDP port 1414, if possible, to 31414. This is, of course, no problem for Skype - the procedure described above continues to work in a similar manner without limitations.
It becomes a little more complicated if a firewall simply assigns ports in sequence, like Check Point's FireWall-1: the first connection is assigned 30001, the next 30002, etc. The Skype server knows that Bob is talking to it from port 31234, but the connection to Alice will run via a different port. But even here Skype is able to outwit the firewall. It simply runs through the ports above 31234 in sequence, hoping at some point to stumble on the right one. But if this doesn't work first go, Skype doesn't give up. Bob's Skype opens a new connection to the Skype server, the source port of which is then used for a further sequence of probes.

Skype can do port scans. Here it succeeds on port 38901 and connects through the firewall.

Nevertheless, in very active networks Alice may not find the correct, open port. The same also applies for a particular type of firewall, which assigns every new connection to a random source port. The Skype server is then unable to tell Alice where to look for a suitable hole in Bob's firewall.
However, even then, Skype doesn't give up. In such cases a Skype server is then used as a relay. It accepts incoming connections from both Alice and Bob and relays the packets onwards. This solution is always possible, as long as the firewall permits outgoing UDP traffic. It involves, however, an additional load on the infrastructure, because all audio data has to run through Skype's servers. The extended packet transmission times can also result in an unpleasant delay.
Use of the procedure described above is not limited to Skype and is known as "UDP hole punching". Other network services such as the Hamachi gaming VPN application, which relies on peer-to-peer communication between computers behind firewalls, use similar procedures. A more developed form has even made it to the rank of a standard - RFC 3489 "Simple Traversal of UDP through NAT" (STUN) describes a protocol which with two STUN clients can get around the restrictions of NAT with the help of a STUN server in many cases. The draft Traversal Using Relay NAT (TURN) protocol describes a possible standard for relay servers.

DIY hole punching

With a few small utilities, you can try out UDP hole punching for yourself. The tools required, hping2 and netcat, can be found in most Linux distributions. Local is a computer behind a Linux firewall (local-fw) with a stateful firewall which only permits outgoing (UDP) connections. For simplicity, in our test the test computer remote was connected directly to the internet with no firewall.
Firstly start a UDP listener on UDP port 14141 on the local/1 console behind the firewall:

local/1# nc -u -l -p 14141

An external computer "remote" then attempts to contact it.

remote# echo "hello" | nc -p 53 -u local-fw 14141

However, as expected nothing is received on local/1 and, thanks to the firewall, nothing is returned to remote. Now on a second console, local/2, hping2, our universal tool for generating IP packets, punches a hole in the firewall:

local/2# hping2 -c 1 -2 -s 14141 -p 53 remote

As long as remote is behaving itself, it will send back a "port unreachable" response via ICMP - however this is of no consequence. On the second attempt

remote# echo "hello" | nc -p 53 -u local-fw 14141

the netcat listener on console local/1 then coughs up a "hello" - the UDP packet from outside has passed through the firewall and arrived at the computer behind it.
Network administrators who do not appreciate this sort of hole in their firewall and are worried about abuse, are left with only one option - they have to block outgoing UDP traffic, or limit it to essential individual cases. UDP is not required for normal internet communication anyway - the web, e-mail and suchlike all use TCP. Streaming protocols may, however, encounter problems, as they often use UDP because of the reduced overhead.
Astonishingly, hole punching also works with TCP. After an outgoing SYN packet the firewall / NAT router will forward incoming packets with suitable IP addresses and ports to the LAN even if they fail to confirm, or confirm the wrong sequence number (ACK). Linux firewalls at least, clearly fail to evaluate this information consistently. Establishing a TCP connection in this way is, however, not quite so simple, because Alice does not have the sequence number sent in Bob's first packet. The packet containing this information was discarded by her firewall.

WebRTC - bringing real time communications to the web.

WebRTC - an open real-time communications project

WebRTC offers web application developers the ability to write rich, realtime multimedia applications (think video chat) on the web, without requiring plugins, downloads or installs. It's purpose is to help build a strong RTC platform that works across multiple web browsers, across multiple platforms.


https://sites.google.com/site/webrtc/blog

Low Level Virtual Machine (LLVM)

LLVM is a really interesting alternative to java and it's bytecode jvm.
What's interesting is it can be run in the web browsers Javascript enterpreter allow this to run on web site.

So you can compile C & C++ code to run on the web in a javascript VM.

> Deepak  - Apple is funding the project.llvm-gcc is the default compiler which is shipped with Mac OS and IOS SDK.


http://llvm.org/

Low Level Virtual Machine (LLVM) is a compiler infrastructure, written in C++, which is designed for compile-time, link-time, run-time, and "idle-time" optimization of programs written in arbitrary programming languages. Originally implemented for C/C++, the language-independent design (and the success) of LLVM has since spawned a wide variety of front ends, including Objective-C, Fortran, Ada, Haskell, Java bytecode, Python, Ruby, ActionScript, GLSL, and others.

Emscripten is an LLVM-to-JavaScript compiler. It takes LLVM bitcode (which can be generated from C/C++, using llvm-gcc or clang, or any other language that can be converted into LLVM) and compiles that into JavaScript, which can be run on the web (or anywhere else JavaScript can run).

From Slashdot: Doom Ported To the Web

"Ever since Id Software released the Doom source code under the GPL, it's been ported to platform after platform. Now, you can play Doom compiled to JavaScript on the web, using standard web technologies like Canvas and without any plugins. If your browser has trouble running it, here's a screencast." The translation was accomplished using Emscripten, a Javascript backend for LLVM. As per the GPL, full source code is available. Pretty neat.

Use the PS3 for your project?

http://www.freebsd.org/news/status/report-2010-07-2010-09.html#FreeBSD-on-the-Playstation-3

Apple TV hacked and turned in to web server.

An Apple TV-Based Webserver - Slashdot

Lua and embedded programming.

Designing a Programming Language For Embeddability - Slashdot

Browse the web from the US

I friend of my is in the Middle east and wishes to browse the web from the US.

I am going to try to setup open VPN and tinyproxy on my Virtual Server Co-Lo for him and see how that works.

https://banu.com/tinyproxy/

http://openvpn.net/index.php/open-source.html

Bash: Find the Max in a list of Integers

Max is a little shells script that will find the maximum value and returns that value and the line it was found at.

max

#!/bin/bash
MAX=0
ELE=0
while read line; do
ELE=$[ $ELE + 1 ]
if [ "$line" -gt "$MAX" ]; then
MAX=$line
POS=$ELE
fi
done
echo "$MAX $POS"


Usage Example

-bash-3.2$ cat ttt
1000000001
1000000000002
1000000000007
1000000000033
9999
1000000000000
1000000000005
-bash-3.2$ ./max < ttt
1000000000033 4
-bash-3.2$


Hackers Find New Way To Cheat On Wall Street

From Slashdot:

"The high-speed trading exchanges that conduct the business of buying and selling stocks and mutual funds are so fast that hackers can introduce delays of a few microseconds completely unnoticed by today's network monitoring technology — and manipulate prices in the process to reap millions of dollars to the detriment of everyone else, InfoWorld's Bill Snyder reports. This kind of activity creates new reason to distrust Wall Street and shows how the computer networks we all rely on for conducting business and moving information are ripe for undetectable hacking."

Preserving Great Tech For Posterity — the 6502

From Slashdot:

"For great old hardware products like the MOS 6502 (used in the Apple II, the C64, the Nintendo NES), the details of the designs have been lost or forgotten. While there have been great efforts to reverse engineer the 6502 from the outside, there has not been the hardware equivalent of the source code — until now. As Russell Cox states: 'A team of three people accumulated a bunch of 6502 chips, applied sulfuric acid to them to strip the casing and expose the actual chips, used a high-resolution photomicroscope to scan the chips, applied computer graphics techniques to build a vector representation of the chip, and finally derived from the vector form what amounts to the circuit diagram of the chip: a list of all 3,510 transistors with inputs, outputs, and what they're connected to. Combining that with a fairly generic (and, as these things go, trivial) "transistor circuit" simulator written in JavaScript and some HTML5 goodness, they created an animated 6502 web page that lets you watch the voltages race around the chip as it executes. For more, see their web site visual6502.org.'"

Update All Scripts in One Swoop

First this is old hat for most admins, so they will sigh a collective yawn.

Now, let's say, like me you have a couple of dozen scripts written in wish8.4 and you want to update all the scripts at once to wish8.5.

sed -i 's/wish8.4/wish8.5' *

• sed - the classic stream editor in Unix (yes even linux and windows have it)
  
• -i - option, edit "in-place", meaning don't output the result, just change silently
  
• s /x/y/ - substitute y for x
• * - command line glob

NOTE the last was for LINUX on BSD use -i .bak 's/....
(added 2011-01-30)

tc

Putin Orders Russian Move To GNU/Linux

From Slashdot:

"Vladimir Putin has signed an order calling for Russian federal authorities to move to GNU/Linux, and for the creation of 'a single repository of free software used in the federal bodies of executive power.' There have been a number of Russian projects to roll out free software, notably in the educational sector, but none so far has really taken off. With the backing of Putin, could this be the breakthrough free software has been waiting for?"

Here is another link to the story:
http://mashable.com/2010/12/27/vladimir-putin-free-software-by-2015/

BSD Philosophy

BSD Unix started at Berkeley University shortly after a very turbulent and rebellious time at Berkeley in the late 60's.
During that time there were several major movements happening all at once. The Free Speech, Anti-War, Hippies, Free City, think for yourself, Acid test, civil rights, and a tornado of other forces that lead to several all out riots on campus.
Other influences at Berkeley are a large Asian Population and the study of eastern cultures (Chinese, Japanese and India) and Philosophy.

Many of these memes later incorporated themselves into the BSD Unix culture and Philosophy.

Respect for elders, New isn't always better


Unix and BSD Unix are old operating systems by our modern perception. But, if you were to think this was a bad thing you'd be totally wrong.


At the root of most Modern Operating Systems are design concepts first pioneered in Unix and BSD. Many way DOS, Windows NT,2000,XP, Mac OS X, NeXT OS, operate are based on UNIX concepts.

Change isn't the same as improvement

"The Great Wall of China wasn't built by swapping any bricks in and out" - John Sokol.

"The devil you know is better than the one you don't"

Unix development started more than three decades ago, and the versions of operating system available today is the product of thousands upon thousands of developers hard work and constant improvements. Along the way, Unix has pioneered some very powerful design concepts and some elgant solutions to hard problems common in computing.

"If I had more time, I would have written a shorter letter." - Mark Twain


"I have made this letter longer than usual, because I lack the time to make it short" (Je n'ai fait celle-ci plus longue parceque je n'ai pas eu le loisir de la faire plus courte) - Blaise Pascal, Lettres Provinciales (1656-1657), no. 16.

"Anyone can make something complex, but it takes a genius to make it simple" - Albert Einstein

Elimination of unnecessary complexity is a Key part of the BSD Philosophy

Much of the "Unix way" is based on the idea of abstracting interfaces into the simplest possible terms, BSD trys to take this to it's limits.

Initially, this was simply because the OS was considered experimental, and the simplest possible interface, one that wouldn't change later, .was the most easiest to construct.
This spend time upfront to think of how to "do it the easy way" methodology has evolved into a many of the most powerful design concepts in OS design. Abstraction that started in Unix is the core of many parts of a modern OS. From Files, Devices, Sockets, Modules, Processes, Memory Management, File Systems, directories, and Pipes.

Even the Internet is based on BSD technology and concepts. The BSD 4.2 OS was the first to Implement TCP/IP and part of that was the concept of network Sockets and Ports. Many other more complex network technology had been around for a while, but BSD Sockets allow for two computers to communicate in a way that uses the exact same commands as file access. Open, Read, Write, Close. It also uses another concept, Streams.


Simplicity and elegance though consistent software interfaces

Where:

elegant : adj. [common; from mathematical usage] Combining simplicity, power, and a certain ineffable grace of design. Higher praise than `clever', `winning', or even cuspy.

"The challenge is keeping it simple while providing the people with functions. This is harder to do. Anyone can make something complex." Dauphin A German Chair company

Try to understand the intent of the original authors' style and intent, and make changes consistent with this


This means not just going in slashing and hacking changes into the Kernel or User space code without researching and thinking them through first. Find a clean way to do what ever your trying to do. Spaghetti code is not tolerated.


All too often a quick and dirty little hack that was meant to be temporary become permanent and hard to fix later.

KISS, Keep It Simple, Stupid or maybe Keep it Short and Simple, I like the first one.

Working on BSD code is basically working on other people code for the most part. Most likely what ever you do, someone else will have to work on after you.

Working on someone else code is something many application programmers realy suck at. Also they think using the latest cool tricks that just came out in the latest language spec or the new compiler supports it so it's really Cool.
F*ck them! These guys need to be slapped up side the head a few times.
I see time and time again the Linux guys keep making a mess for those who need to update there code and drivers over to the latest OS version. Often is is almost as bad as a cross platform port, nothing works out the box everything is broken and it becomes a major undertaking!

The Mac OS-X darwin they are mixing C++ and Objective C into the kernel source tree! It's full of broken locks and is nearly impossible to understand anymore. Leaving some messy and hard kernel hacking to the customers having to make there code work on a pig of an OS. I hope that doesn't sound like I'm venting too much, does it?


"A program with a colorful GUI is like a woman with too much makeup. Shiny on the outside - awful on the inside."

Quote from the NET



Are you going to run a gui or from the command line etc... over SSH?
If you want a gui I would go Linux maybe Fedora, for SSH use FreeBSD...I used to be against it but I am now a convert to the Church of BSD.

Why would I ever want a GUI? It may sound odd but I have never ran X on any of the 100's of FreeBSD systems I have worked on. Command line Interfaces Rull, and for most of the stuff I do is much faster. Even this web page is edited in "VI".


We are not Linux, We were here before them; and well be here after them

So what's the difference, much of the same code get's passed between BSD and Linux Right? That's only partly true. The Licenses are incompatable between the two for starters.

"I must say the linux community is a lot nicer than the unix
community. a negative comment on unix would warrant death
threats. With linux, it is like stirring up a nest of butterflies."
             -- Ken Thompson author of C Language. 1999 

BSD vs. Linux, Any questions?

FreeBSD, OpenBSD and SuSE 6.2 Eval Review by Keith Rankin:
"The differences between FreeBSD and Linux used to be much more obvious than they are now.
Now it comes down to theology. The BSD world is still the 'high church' or Druid Unix. Blood will be spilled on a stone altar at midnite when star systems are in a certain alignment to learn the ways of this tribe. Linux is a happier world. The spirits of Captain Kirk, Peter Pan and good beer come to mind."

Hackers and the bushido code

For much of the early life of BSD Unix was spent running campus MiniComputers. PDP/11, Vax 11/750 and the like.

One way to attain elite status (enlightenment or become the master or Guru) was to "break root", break in or even just be able to crash the server.

This was not viewed as a federal offense, or even as something inappropriate but as an exciting challenge and testing the limits of the box.

For the many years it was an arms race between the OS developers and students to see if the box could be crashed by malicoius code or commands, the end result is something rock solid.

Now crashing a windows box usualy requires little more then power it up, but most BSD systems can go for years without rebooting, crashing or having it security compromised. Even on my own servers, like the one that ran the LeonardoDicaprio.com site, despite countless attempts to take out the box, it had little effect.

Part of Hacker Philosophy directly stems from these early BSD days, although not all of it. One of the thing was to not damage or sabotage, but to report and inform people of the bug or security hole. Maybe a little Kilroy was here on the administrators console ;)

Reboots, re-compiles and Re/Installations are EVIL and to be avoided at all costs.


One important thing about BSD is the belief that you should be able to stress the system in any number of ways, like open up and infinite number of files, sockets, programs, memory, or any other resource and have the system respond gracefully. But this I mean not to have a Blue Screen of Death, or Kernel Panic.


Linux groups have installfests, while BSD groups have installathons. Clearly an installfests sounds festive and much happier then an installathons that is more like a Marathon to try to race through it.

We are the one true Path, all non-believers (can go to hell) are damned to hell

This is in the tradition of all great religions.

From: Nick Moffitt nick@zork.net , With some paraphrasing.


Tue, 22 Aug 2000 14:02:47
Basically, user groups nowadays are either:

 Lame talking-heads lecture series.

 Cowtow to the windows users installathons.


  I say that's bullsh*t, and we all know it. F*CK the newbies.

We'll hold installfests, sure, but we'll be perfectly justified in telling people to take their 386s with 15 year-old RLL drives and GO HOME. This will be a meeting of COOL PEOPLE to show off COOL STUFF!

We'll not be limited to a particular OS, either. We'll have Linux weenies, FreeBSD weenies, and cypherpunk weenies clutching their precious OpenBSD! We'll have the AMAZING GEORGE PERRY and his BEARDFUL OF FORTH!

I think that expresses the attitude fairly well...

BSD License

Do what is right and be hated by those around you. Do what is popular and be hated by history.
The BSD License allows anyone to use the code for any reason and any way with no restrictions other then credit is given to the Authors
Transparency Begets Trust - Expertise in niches, transparency in motives and thought process and owning up to mistakes publicly create a trust relationship" - Will Pate



In General people in the BSD world view people and Individuals who do not share code and improvements as pompous idiots. There are a ton of smart people in the BSD world and no one individual or company (M$) not sharing code is really going to make any difference, the reality is we probably wouldn't want there code anyhow.

Trust builds Confidence

They will loose the benefits of having 1000's of eyes screening the code for holes and bugs as well as tightening things up and making improvements.

This is very much same philosophy as the Private vs. Public research in almost any scientific field. If you keep your research to your self, then you don't get credited for it. Also your work never gets confirmed, there is a good chance of deluding yourself about how secure and bug free your code is. (take Microsoft for an example)

The BSD approach is consider by most people to be better for businesses, companies such as Apple and Microsoft benefited their users by incorporating BSD-licensed code without having to share there code or changes to this code.

When questioned, most GPL programmers did not want the work they released to be used in a manner they did not support.


The BSD philosophy seems to hold that creating and giving away code, then seeing it used by others, is victory and reward enough.


But most of the GPL supporters disapproved of allowing "others" to close off source code and hide enhancements.

Why the Church of BSD

BSD has been through many Holy Wars and has been called a religion by many people for a long time. Even it's experts are called Guru's , High Priests and Wizards. What BSD programmers do has sometimes been called VooDoo, Alchemy and Black Magic. People who violate our way of doing things are declared Heretics, and publicly flamed! We even have our own official canonical way of doing things and this is in the source code.

The Source code is our Bible.

It seems we are long over due for a formal Church of BSD.

Canonical is an adjective derived from canon. It essentially means "standard", "generally accepted" or "part of the back-story." Canonical in my context means reduced to the simplest and most significant form possible without loss of generality.
This word is usualy used by theologians and canon lawyers to refer to the canons of the Eastern Orthodox and Roman Catholic churches, adopted by ecumenical councils.

Hacker Slang - Holy Wars From answers.com
[from Usenet, but may predate it; common] n. flame wars over religious issues. The paper by Danny Cohen that popularized the terms big-endian and little-endian in connection with the LSB-first/MSB-first controversy was entitled On Holy Wars and a Plea for Peace. Great holy wars of the past have included ITS vs. Unix, Unix vs. VMS, BSD Unix vs. System V, C vs. Pascal, C vs. FORTRAN, etc. In the year 2003, popular favorites of the day are KDE vs. GNOME, vim vs. elvis, Linux vs. [Free|Net|Open]BSD. Hardy perennials include EMACS vs. VI, my personal computer vs. everyone else's personal computer, ad nauseam. The characteristic that distinguishes holy wars from normal technical disputes is that in a holy war most of the participants spend their time trying to pass off personal value choices and cultural attachments as objective technical evaluations. This happens precisely because in a true holy war, the actual substantive differences between the sides are relatively minor. See also theology.

What about Chuck?

The BSD Logo.

One another occasion a very christian individual was noticing the devil logo, I tried to explain it was a daemon, not demon or devil and didn't have any satanic meaning, again they freaked.

You have to admit, that's sort of a Satanic symbol if you ever saw one, right?

From: FreeBSD.com
"Many people equate the word ``daemon'' with the word ``demon,'' implying some kind of Satanic connection between UNIX and the underworld. This is an egregious misunderstanding. ``Daemon'' is actually a much older form of ``demon''; daemons have no particular bias towards good or evil, but rather serve to help define a person's character or personality. The ancient Greeks' concept of a ``personal daemon'' was similar to the modern concept of a ``guardian angel'' --- ``eudaemonia'' is the state of being helped or protected by a kindly spirit. As a rule, UNIX systems seem to be infested with both daemons and demons." (p403) The earliest (and most popular) renditions of the BSD Daemon were created by John Lasseter.

That cute little red guy with the pitch fork is named Chuck

First Drawing with the Daemon Logo

Dictionary.com defines Daemon as A program or process that sits idly in the background until it is invoked to perform its task.

Historicaly though I always though of Chuck as more of a Gremlin really very much like the one portrayed in a Bugs Bunny cartoon "Falling Hare (1943)" before it was restored.

Officially Gremlins started in the 1920 at a RAF term for a low-ranking man saddled with oppressive assignments.

Gus the Airplane Military Insignia

Although this terms really take on color in World War II as a "an imaginary gnomelike creature who causes difficulties in aircraft."

Military Airplane Insignias

Also the Irish Gaelic word gruaimin, "ill-humored little fellow."

Drawing from Roald Dahl's book, "The Gremlins" Front Cover

This very much conjures up visions of these little gremlins running about secretly doing things.

From Roald Dahl's book, "The Gremlins"
"I've just found a nest of widgets," he said, and looked around in triumph. "Widgets?" said Stuffy. "Widgets? Never heard of them!"
But Jamface went on. "As I said, I've just found a nest of widgets located in the rear turret of my plane. There were twelve of them . very young ones."
"What are widgets?" asked Gus.
"It's very simple," said Jamface. "Widgets are the young of gremlins and fifinellas. No one knows until they grow up whether they are going to turn into males or females, but it's usually males; in each nest of twelve widgets only one will eventually turn into a fifinella."

The Church of BSD: Background

We are not a Satanic Cult Although we may be a cult, this has yet to be determined

One day I was at a restaurant explaining process control to one of my disciples.
I was mentioning how we have to kill the children (child processes) if they become unresponsive. Or we can even set an alarm for the children to kill themselves. That the parent need to wait (wait3) and acknowledge that the child has died or else it will become a zombie. The look of horror the woman sitting across had was unforgettable. I tried to explain it was a computer software thing but it was too late, she fled terrified, probably to call the police or something. I didn't really want to stick around too long to find out.
From: man ps Z Marks a dead process (a ``zombie'').

Hey, you just can't make up shit like this.

Don't forget that you have to Fork to Spawn Children, and the only way to Kill

On another occasion while being filmed doing a live video streaming event with Arthur C. Clarke, The camera filmed me key flying across the keyboard typing "kill 666". Just sort of odd the process ID, I didn't think much of it till It ended up in the final cut of a documentary video made about it.

And what about "chmod 666".

They say when you play a Microsoft CD backwards you can hear satanic messages...but that's nothing, if you play it forward it will install Windows! 
 
And what about Chuck?

 One another occasion a very christian individual was noticing the devil logo, I tried to explain it was a daemon, not demon or devil and didn't have any satanic meaning, again they freaked.

You have to admit, that's sort of a Satanic symbol if you ever saw one, right?

Read more at "What about Chuck".

BSD Unix History

Unix Started in AT&T Bell Labs in 1969. At time time is was freely shared as "scrap" The Berkely BSD 4.2 1983 OS was the first OS to support TCP/IP.
The Unix wars were the struggles between vendors of the Unix computer operating system in the late 1980s and early 1990s to set the standard for Unix henceforth. These battles are commonly held to have harmed the market acceptance of Unix and created a market gap that allowed the rise of Windows NT.

In the mid-1980s, the two common versions of Unix were BSD, from the University of California at Berkeley BSD4.2 released in 1983 , and System V, from AT&T. Both were derived from the earlier Version 7 Unix, but had diverged considerably. (This conflict was also known as the "UNIX wars" to some degree.) Further, each vendor's version of Unix was different to a greater or lesser degree.


A group of vendors formed the X/Open standards group in 1984, with the aim of forming compatible open systems. They chose to base their system on Unix.


X/Open caught AT&T's attention. To increase the uniformity of Unix, AT&T and leading BSD Unix vendor Sun Microsystems started work in 1987 on a unified system. This was eventually released as System V Release 4 (SVR4).

While this decision was applauded by customers and the trade press, other Unix licensees feared Sun would be unduly advantaged. They formed the Open Software Foundation (OSF), who released OSF/1, more closely based on BSD. AT&T and another group of licensees then formed UNIX International.

Technical issues soon took a back seat to vicious and public commercial competition between the two competing "open" versions of Unix, with X/Open holding the middle ground.

Linux kernel based on rewrite of Minux was released to the Internet in September 1991,

The first open source Unix port began in 1989 with 386BSD and first, incomplete traces of the port can be found in 4.3BSD NET/2 of 1991. It was first released in March 1992 (version 0.0) and in a much more usable version on July 14, 1992 (version 0.1).

Due to a lawsuit by AT&T, USL v. BSDi, some potentially so-called encumbered sources which existed within 386BSD were to be removed from all the derived systems, and the distribution of 386BSD was to be stopped. In late 1993


In 1993, AT&T sold Unix to Novell, who assigned trademark rights to X/Open. In 1996, X/Open and the OSF merged to form the Open Group, whose Single UNIX Specification is now the single standard for proprietary Unix. However, the damage to Unix's market share had been done.

BSD Unix

The BSD varients (FreeBSD, NetBSD, OpenBSD, etc) are UNIX based operating systems
that work on Intel PC based systems that have a reputation for being highly optimised,
highly secure and excellent (when decently configured) for keeping hackers at bay.


Linux does not compare all that well to BSD: having a reputation of being
a hacker's paradise to break into compared to BSD (FreeBSD at least).
FreeBSD (and the other BSD's) run some of the most intensely hit servers
on the globe, including ftp://ftp.cdrom.com. It can be said that the nicest
thing you can do for a hacker is replace Windows with Linux as not only is it
potentially easier to hack (one of the benefits of open source), but the system
becomes worth hacking into. The same cannot necessarily be said with BSD based UNIX's
which are built of sterner stuff to keep hackers at bay.


Thus the BSD based operating systems of which FreeBSD is one - have a reputation of being
better designed (by gurus no less!), more secure (and more security
conscious) and more robust than most other operating systems (including
Linux - an operating system - it is claimed - "written by kiddies - for kiddies").
Refer to:
FreeBSD, OpenBSD and SuSE 6.2 Eval Review by Keith Rankin: "The differences between
FreeBSD and Linux used to be much more obvious than they are now.
Now it comes down to theology. The BSD world is still the 'high church' or Druid Unix. Blood will
be spilled on a stone altar at midnite when star systems are in
a certain alignment to learn the ways of this tribe. Linux is a
happier world. The spirits of Captain Kirk, Peter Pan and good beer come to mind."

("Buckets of Blood" and non-alcoholic prudery is official CCP14 computer operating system policy)

Remove all but the newest packages on hosts

Remove all but the newest packages on hosts

for h in $(ls XXX* | sed 's/\(\w*\)\..*/\1/' | sort -u ); do

ls -t $h*  | tail -n +4 

done | xargs sudo rm

list just packages.

sed 's/\(\w*\)\..*/\1/'  

this turns "XXXyyy-1.23.rpm" in to XXXyyy-1

sort -u 

then only give me unique package names.

for h

indexs through each type of package.

Then we pass each package type $h* in to a ls -t that sorts by time,

tail -n +4  

this removes the first 3 lines from what it list in the ls, just the 3 newest versions

xargs 

then takes this list of files from everything before and passes it as an argument to rm

After IPv4, How Will the Internet Function?

from Slashdot:

"36 countries in the world have over 100% per-capita usage of mobile phones, and this is driving a real crunch on IPv4 addresses as more and more of these devices are data-capable. The mobile network operators are acting fast to deploy IPv6, and T-Mobile USA has had an IPv6-only trial going on for over 9 months now using NAT64 to bridge to IPv4 Internet content. It is interesting to note that the original plan for IPv6 transition, dual-stack, has failed since IPv4 addresses are effectively already exhausted for many people who want them. Dual-stack also causes many other issues and has forced the IETF to generate workarounds for end users called happy eyeballs (implying that eyeballs are not happy with dual-stack), and a big stink around DNS white-listing. How will you ensure that your network, users, and services continue to work in the address-fractured world of the future where some users have only IPv4 (AT&T ), some users have only IPv6 (mobile and machine-to-machine as well as developing countries), and other Internet nodes have both?"

Progress In Algorithms Beats Moore's Law

"Seen on the blog 'Algorithmic Game Theory,' a report to congress and the president about past and future advances in information technology notes that, while improvements in hardware accounted for an approximate 1,000 fold increase in calculation speed over a 15-year time-span, improvements in algorithms accounted for an over 43,000 fold increase."

Ubuntu blocking X11 solution

My servers are OpenSuSe and my personal machine is Ubuntu. On a local area network, I can run my X11 editor and edit files on the server. But remotely, like from home, it was not clear.

This did it. Add to /etc/ssh/ssh_config

ClearAllForwardings yes

I also explicitly set the ssh server

X11Forwarding yes

it is not clear if this is the default


Today (2010-12-29), I also noted in /etc/ssh/ssh_config

ForwardX11 yes

tc

The Mysteries of the Unix Date Command.

To do any math on dates is very difficult in any language. But in the Unix/Linux shell this can be a breeze.

This is done by converting all data time in to Seconds.

In Unix this is done by counting seconds from UTC or Universal Constant Time, 1/1/1970

So once you convert in to second, you can do math as usual, then convert time back to date time.


Example of how to convert back and forth

Conversion from Seconds UTC to string

-bash-3.2$ date --date "1970-01-01 1292970890 sec utc"
Tue Dec 21 14:34:50 PST 2010

Conversion from string to Seconds UTC

-bash-3.2$ date --date "12/21/2010 14:34:50 PST" "+%s"
1292970890
-bash-3.2$ date --date "Dec 21 14:34:50 PST 2010" "+%s"
1292970890

Current time

-bash-3.2$ date "+%s"
1292979451

Get Seconds UTC file creation date.

stat -c %Y filename

ls -l
-rw-r--r-- 1 jsokol jsokol       8 Sep 22 18:36 t
 

-bash-3.2$ stat -c %Y t
1285205797

So for doing math on time:

Get file age

(In Bash)
Returns how many hours old

expr \( `date +%s` - `stat -c %Z $filename` \) / 3600

Return age of oldest file in directory

expr \( `date +%s` - `stat -c %Y \`ls -t | tail -1\` ` \) / 3600

Create Shell alias of this

alias oldest='expr \( `date +%s` - `stat -c %Y \`ls -t | tail -1\` ` \) / 3600'

bash process ID PID

$$, My PID
$!, My Child PID
$PPID, My Parent PID

bash command line

#!/bin/bash
echo "$*"
echo "$@"
echo "$_"
echo "$#"

./test a b c
a b c
a b c
c
3

Ansi color codes in Ruby


#!/usr/bin/ruby
esc="\e["
st=""
x=30
 while x < 38 do
   y=40
   while y < 48 do
     st +=  esc+"#{x};#{y}m #{x};#{y} "
     y+=1
   end
   puts st+esc+"0m\a"
   st=""
   x += 1
 end
puts esc+"0m"



This output's



With a little work I am sure this can be made in to a simple function or library.
or just something you can send with a puts.

Ruby to MBox format.

I have over the past 20+ years written many scripts to deal with incoming mail.
I can process statistics from machine generated alert E-mails, rebroadcast E-mails and generate pages from them based on there content.


I was using a the IMAPClient library in Perl for this, but when my IMAP Mail server was migrate to an SSL based TLS/IMAP everything broke.

TLS encrypted port 993 RFC2595 RFC3501

After many long hours of struggling to get IMAPClient to work over SSL I just had to give up. It looks like something is just broken in that lib, I tried opening my own SSL socket then passing it to the lib, I tried having the lib itself connect to the SSL socket, no luck.

The Ruby net/imap just seems better written and it does work. Unfortunately it's lacking the message_to_file  function which I had to replace that functionality.



A small change in the scripts below could also simulate .forward also or support mail dir.
If you do that, please send me a copy  


In Perl for Non-secure IMAP

#!/usr/bin/perl

use Mail::IMAPClient;
my $imap = Mail::IMAPClient->new( Server       => 'mail.yourdomain:143',
                                   User         => 'jsokol',
                                   Password     => 'yea-right')
        or die "IMAP Failure: $@";

 foreach my $box qw( alarmpoint ) {
   my $file = "/uhome/jsokol/crapmail/". $box;
   $imap->select($box);

   my @msgs = $imap->search('ALL')
        or die "Couldn't get all messages\n";

   foreach my $msg (@msgs) {
     open my $pipe, "| formail >> $file"
       or die("Formail Open Pipe Error: $!");
     $imap->message_to_file($pipe, $msg);
     close $pipe
       or die("Formail Close Pipe Error: $!");
     $imap->delete_message($msg);
   }

   # Now expunge the messages and close the folder
   $imap->expunge($box);
   $imap->close($box);
 }

 $imap->logout();

I am just learning Ruby, so please excuse some of this code.

In Ruby secure TLS/IMAP

#!/usr/bin/ruby
require 'net/imap'

        server = "mail.yourdomain.com"
        port = 993
        ssl = true
        username = "jsokol"
        password = "yea-right"


        Net::IMAP.debug = false
        conn = Net::IMAP.new(server, port, ssl)
        resp = conn.login(username, password)

        conn.select('test')
        conn.search(['ALL']).each do |sequence|

                env = conn.fetch( sequence,"ENVELOPE")[0].attr['ENVELOPE']
                xdate =  env.date.split(" ");

               wkday = xdate[0].sub(/,/,"")
               day = xdate[1]
               month = xdate[2]
               year = xdate[3]
               time = xdate[4]


               if (day.length < 2)
                   day = "0" + day
               end

                box  = env.from[0].mailbox  + "@" +  env.from[0].host + "  " + wkday + " " + month + " " + day + " " + time + " " + year

                fetch_result = conn.fetch( sequence,  "RFC822")[0].attr['RFC822']
                if fetch_result
                     File.open( "MAILOUT" , "a") {|file|  file.write ("From #{box}\n#{fetch_result}\n\n\n") }
                end

                conn.store(sequence , "+FLAGS", [:Deleted]) # remove this line if you don't want to deltete these E-mails.
        end
        conn.expunge # remove this line if you don't want to deltete these E-mails.
        conn.logout
        conn.disconnect

Google Public DNS

Few people are aware of this but Google offers a DNS service.

This is particularly useful in a number of situations.

*If you are inside a corporate firewall and want to test host DNS entries appear to the outside world.

*If you don't trust your local ISP, or their DNS is having issues.

I often run scripts to check massive blocks of IP's

#while read ADDR
for ADDR in $IPLIST; do
 DNSR=`host $ADDR 8.8.8.8 | tail -1 | awk ' {print $NF}' | sed 's/3(NXDOMAIN)/NONE/'

echo  "$HOST : $ADDR $DNSR" >> ADDR-LIST.txt

done

Google Public DNS IP addresses

The Google Public DNS IP addresses are as follows:
8.8.8.8
8.8.4.4

Using Google Public DNS

8.8.8.8, A DNS Number for Faster Browsing

Lua Programming

Lua is a sexy little language out of Brazil.


http://wiki.eluaproject.net/Overview

http://wiki.eluaproject.net/

Ruby programming

Been learning Ruby lately. Didn't think I'd like it better then Perl But I do.

I will post some links I have found useful on this page.

Ruby Basic Tutorial

Ruby in Twenty Minutes

Ruby in 100 Minutes

Ruby 101: Substrings in Ruby

Some files space is just wasted

Recently, while testing some software that did directory tree displays, I noted 1,505,297 bytes used just to store the names for the /usr directory for Linux. Or 28000+ subdirectories.

Again, that is 1,505,297 bytes just for the directory names.

tc

Oil and Lube

One would think that computers are not in need of regular maintenance, in fact they are. After some time in operations - months or years, any original lubricants are gone. Recently, SVBUG was gifted several servers that had seen their prime, but were still useful. A little Oil and Lube did the trick.

Here are a couple of Tutorials on :

How to Remove and Apply Thermal Compound By Jeremy Blum (aka sciguy14)
http://www.youtube.com/watch?v=I3gx6c62D7I

Quiet noisy computer fans with a drop of oil by Greg Shultz
http://articles.techrepublic.com.com/5100-10878_11-5034842.html

tc

BSD Event Feed and RSSFeed

Looks like a semi-official BSD Events list,

http://twitter.com/bsdevents

The RSS Feed is on the right below the menu junk. It's also below:

http://twitter.com/statuses/user_timeline/23451058.rss

tc

Super Computer taking over?

In a recent story (May, 2010), SEC Chairman Admits: We’re Outgunned By Market Supercomputers, a luddite might find reason to fear big brother. But in fact a mystery remains from the so-called "Fat Finger Incident" on May 6, 2010. The incident, such that it was, trigger "a total drop of 9.16 percent from the previous day’s close".

I'm sure conspiracy theories will abound as to the reason for this - including the demise of certain individuals. More to the point of this article, it cannot be sufficiently underestimated the problems the stock market has created.

It has been reported that because modern stock trading is essential completely electronic --- that dealers look to have their computers as close to the trading computers as possible. The idea is to be able to process orders microseconds quicker than the competition.

In fact it has also been reported that traders are "snipping" as the market moves. That is, "snipping" makes a few cents, if not hundreds of a cent as the market (or a particular stock) moves in a direction. It has even been suggested that dealers are making these "few cents" on their own client's "trades". It has even been suggested that these dealers, having prior knowledge of the limits of their clients, are pushing the pricing to the limit.

This last scenario, as suggested, means the buyer is actually paying more than they would have had the closing price not been known to the dealer.

If this is true, then it appears the dealers are undermining the confidence of their own business.

tc

Notes on Cloud Computing

From a recent BayLisa meeting, in which the speaker failed to arrive, the topic of the night was "Cloud Computing". The first order of business was to define its meanings.

The moderator started by asking if anyone had direct experience in this or similar. I started by stating I had worked in a video-over-the-internet business in the late 90s. Then stated that "cloud computing" was marketing hype.

To make this short, this is the best definition I could write down:

Cloud Computing

Either a high-speed on-demand distributed or a high-speed on-demand clustered computing with invisible redundancy. The best uses to date appear to be optical recognition, and audio recognition. In addition, there are working businesses very reminiscent of "Thin Computing".

Happy sysadmin day everyone!

http://sysadminday.com/

>  What about the DBAs? They need something too:

http://www.dbaday.com/

ORA-24123 feature "dbaday" is not yet implemented

24123, 00000, "feature %s is not yet implemented"
// *Cause: An attempt was made to use the specified feature, but the feature is
// not yet implemented.
// *Action: Do not attempt to use the feature.
// 

BSD Magazine online

http://bsdmag.org/

Video of BSD Conferencs on YouTube

YouTube - Videos of conferences
http://www.youtube.com/bsdconferences

Apps, Apps, How many apps?

While browsing the NetBSD website I found this nice chart.

tc

scp recursive copy

If you've set up a copy using:

scp -r tc@cup:~/myremotestuff/* .

You may have gotten the error

scp: No match

What is not obvious is "what the error is". What has happend is scp tries to match to a local name when infact the intent is to match a remote name. Remove the wildcard (*) and all will be good.

tc

How to write more reliable servers. - dealing with failures

Given enough time most Internet servers will crash. It could be a memory leak, or some unexpected behavior from a new browser, or a deliberate denial of service attack. It could even be from hardware problems such a flaky memory chips.

The question is how do you deal with this.

If you take your basic C/C++ TCP/IP server application, when it dies it dies.
Many people set up some mechanism to monitor it and page a technician to restart it. This has many drawbacks and can lead to periods of prolonged downtime if alerts are missed or crashes become frequent.

Web servers only became reliable when NCSA 1.4.1 came it. It was very well thought out and well written and most web service since have copied the mechanisms it used. NCSA used a parent process that spawn child processes to actually do the work. When a child process dies, the parent wakes up and immediately restarts them.

In my former company IBS(Internet Broadcast Systems) and later at DVBS (Digital Video Broadcast systems) we called this a keep_alive.


The simplest example can be a shells script.

while true
do
server
echo "server crashed" > log
sleep 1
done

With in C server code

int main(int argc, char *argv[])
{

    printf("Started...\n");

...

  Refork:
    switch (pid = fork()) {
    case -1:
 printf("\nCan't fork!\n");
 FatalError(3);
    case 0:
 break;
    default:
 wait(0);
 printf("\nChild died!\n");
 goto Refork;
    }

    printf("Forked...\n");

Linux Disable PC Speaker beeps

Runtime change

• Check if you have the pcspkr module loaded.

#lsmod | grep pcspkr
pcspkr                  7105  0

• Remove the module. lsmod will return nothing if the module was removed.

#rmmod pcspkr
#lsmod | grep pcspkr

• Restore the module when done.

#modprobe pcspkr
#lsmod | grep pcspkr
pcspkr                  7105  0

Permanent change

• Add the pcspkr module to the modprobe blacklist file.

# vi /etc/modprobe.d/blacklist
{Add the lines below to the file.}
# pcspkr - turn off pc speaker "BEEP!"
blacklist pcspkr

• Reboot, and check if the pcspkr module was loaded. If the blacklist file kicked in then nothing will be returned.

Google Chrome

It's using many of the concepts I expored in my talk about the Amorphous OS, I will have put those slides back on the internet.

A big part is no local files.

http://googleblog.blogspot.com/2009/11/releasing-chromium-os-open-source.html

http://www.chromium.org/chromium-os/chromiumos-design-docs

http://en.wikipedia.org/wiki/Google_Chrome_OS

Looks like it's build on a stock Linux Kernel.. I was thinking about this and they need to get ride of the paradigm of a process and replace it. 
Processes have there own time slices and memory.  They seems to have spend a lot of effort to give each browser tab it's own process, and in some circumstances they could not.

What they need to do it get rid of the whole concept and go with an object based memory model.

Where some objects are shared between processes where processes are not sandboxed memory as much as just time slices and sandboxed memory objects associated with parent objects.

An Inside Look At Warhammer Online's Server Setup

From Slashdot: http://games.slashdot.org/story/09/12/31/0428200/An-Inside-Look-At-Warhammer-Onlines-Server-Setup

An article at Gamasutra provides some details on the hardware Mythic uses to power Warhammer Online, courtesy of Chief Technical Officer Matt Shaw and Online Technical Director Andrew Mann. Quoting: "At any given time, approximately 2,000 servers are in operation, supporting the gameplay in WAR. Matt Shaw commented, 'What we call a server to the user, that main server is actually a cluster of a number of machines. Our Server Farm in Virginia, for example,' Mann said, 'has about 60 Dell Blade chassis running Warhammer Online — each hosting up to 16 servers. All in all, we have about 700 servers in operation at this location.' ... 'We use blade architecture heavily for Warhammer Online,' Mann noted. 'Almost every server that we deploy is a blade system. We don't use virtualization; our software is somewhat virtualized itself. We've always had the technology to run our game world across several pieces of hardware. It's application-layer clustering at a process level. Virtualization wouldn't gain us much because we already run very close to peak CPU usage on these systems.' ... The normalized server configuration — in use across all of the Mythic-managed facilities — features dual Quad-Core Intel Xeon processors running at 3 GHz with 8 GB of RAM."

Why Coder Pay Isn't Proportional To Productivity

From Slashdot: http://developers.slashdot.org/story/09/12/23/1820214/Why-Coder-Pay-Isnt-Proportional-To-Productivity

Why Coder Pay Isn't Proportional To Productivity
"John D. Cook takes a stab at explaining why programmers are not paid in proportion to their productivity. The basic problem, Cook explains, is that extreme programmer productivity may not be obvious. A salesman who sells 10x as much as his peers will be noticed, and compensated accordingly. And if a bricklayer were 10x more productive than his peers, this would be obvious too (it doesn't happen). But the best programmers do not write 10x as many lines of code; nor do they work 10x as many hours. Programmers are most effective when they avoid writing code. An über-programmer, Cook explains, is likely to be someone who stares quietly into space and then says 'Hmm. I think I've seen something like this before.'"

The Underhanded C Contest

http://underhanded.xcott.com/

An Open Source Compiler From CUDA To X86-Multicore

From Slashdot:

An Open Source Compiler From CUDA To X86-Multicore

"An open source project, Ocelot, has recently released a just-in-time compiler for CUDA, allowing the same programs to be run on NVIDIA GPUs or x86 CPUs and providing an alternative to OpenCL. A description of the compiler was recently posted on the NVIDIA forums. The compiler works by translating GPU instructions to LLVM and then generating native code for any LLVM target. It has been validated against over 100 CUDA applications. All of the code is available under the New BSD license."

X11 Server Setting No Mouse, Try AllowEmptyInput

Upon setting up FreeBSD 7.2 from 6.4, I though that unrolling the tarball would do it. I did copy xorg.conf to /etc/X11, but no luck. I then read the error log (/var/log/Xorg.0.log). After a search on wiki.x.org, I did see a reference on to how to TURN OFF all keyboard and mouse input by adding to the Section "ServeFlags"

Option "AllowEmptyInput" "True"

I reversed it to

Option "AllowEmptyInput" "False"



tc

Fork Bomb

The bash fork() bomb
:(){ :|:& };:
This is a bash function. It gets called recursively (recursive function). This is most horrible code for any Unix / Linux box. It is often used by sys admin to test user processes limitations (Linux process limits can be configured via /etc/security/limits.conf and PAM).

Once a successful fork bomb has been activated in a system it may not be possible to resume normal operation without rebooting, as the only solution to a fork bomb is to destroy all instances of it.

Understanding :(){ :|:& };: fork() bomb code

:() - It is a function name. It accepts no arguments at all. Generally, bash function is defined as follows:

foo(){
arg1=$1
echo ''
#do_something on $arg argument
}

fork() bomb is defined as follows:

:(){
:|:&
};:

:|: - Next it call itself using programming technique called recursion and pipes the output to another call of the function ':'. The worst part is function get called two times to bomb your system.

& - Puts the function call in the background so child cannot die at all and start eating system resources.

; - Terminate the function definition

: - Call (run) the function aka set the fork() bomb.

OpenDarwin has Shut Down!

When did this happen?   I am finally getting around to developing a mac app. I wanted to use SDL and they had a great ports collection apparently http://darwinports.opendarwin.org/ that is now GONE! Shit.

http://docs.huihoo.com/darwin/opendarwin/news/shutdown.html

OpenDarwin Shutting Down

2006.07.25

OpenDarwin was originally created with the goal of providing a development environment for building and developing Mac OS X sources as well as developing a standalone Darwin OS derivative. OpenDarwin was meant to be a development community and a proving ground for fixes and features for Mac OS X and Darwin, which could be picked up by Apple for inclusion in the canonical sources. OpenDarwin has failed to achieve its goals in 4 years of operation, and moves further from achieving these goals as time goes on. For this reason, OpenDarwin will be shutting down.
Over the past few years, OpenDarwin has become a mere hosting facility for Mac OS X related projects. The original notions of developing the Mac OS X and Darwin sources has not panned out. Availability of sources, interaction with Apple representatives, difficulty building and tracking sources, and a lack of interest from the community have all contributed to this. Administering a system to host other people's projects is not what the remaining OpenDarwin contributors had signed up for and have been doing this thankless task far longer than they expected. It is time for OpenDarwin to go dark.
Project admins for all active projects have been notified, and we will be working with them to provide as seamless a transition to their new homes as possible. We don't want to boot anyone off, we will be operating the machines as usual for several months, until everyone has had a chance to move elsewhere.
We will continue to provide email and dns redirection after the machines go dark. We'll be looking at what other redirection services are needed and can be provided after hosting has ceased.
The OpenDarwin team would like to thank everyone who did contribute to the project, and our apologies to active, loyal projects that have to move.
Thanks,
- OpenDarwin Core Team and Administrators