Showing posts with label shell. Show all posts
Showing posts with label shell. Show all posts

Thursday, May 16, 2013

Countdown Clock/Timer in /bin/bash

Often there is a need to have time between batch jobs. Knowing how much time there is between jobs is useful.

Here are two version.
  1. One I found on unix.com here by cfajohnson. He claims it will work in any POSIX shell
  2. The other I (tc) wrote.

Version #1

# by cfajohnson
countdown()
(
  IFS=:
  set -- $*
  secs=$(( ${1#0} * 3600 + ${2#0} * 60 + ${3#0} ))
  while [ $secs -gt 0 ]
  do
    sleep 1 &
    printf "\r%02d:%02d:%02d" $((secs/3600)) $(( (secs/60)%60)) $((secs%60))
    secs=$(( $secs - 1 ))
    wait
  done
  echo
)

countdown "00:07:55"

Version #2 


# by teacup
countSecondsDown()
{
  seconds=$1
  while [ $seconds -ge 0 ]
  do
    # you need the ampersand after sleep if you use wait.
    # because wait will wait for child process of sleep to complete.
    sleep 1 &
    printf "\r%d  " $seconds
    seconds=$(( $seconds - 1 ))
    wait
  done
  echo

}

countSecondsDown 71
 

Thursday, December 08, 2011

Researchers Expanding Diff, Grep Unix Tools

Researchers Expanding Diff, Grep Unix Tools

Posted by timothy  
from the now-with-raisins dept.
itwbennett writes"At the Usenix Large Installation System Administration (LISA) conference being held this week in Boston, two Dartmouth computer scientists presented variants of the grep and diff Unix command line utilities that can handle more complex types of data. The new programs, called Context-Free Grep and Hierarchical Diff, will provide the ability to parse blocks of data rather than single lines. The research has been funded in part by Google and the U.S. Energy Department."

Friday, November 11, 2011

Generation of Random Numbers from PING

When looking for a flawless Random number generator I realized ping would be it.  This worked well because we were already testing the connectivity between hosts using scripted ping, so we could repurpose that data for random number generation.

We could also save the random numbers to disk and use them one time as a pad then delete them.

sokol@server:~/test$ ./getrand
97598993
sokol@server:~/test$ ./getrand
27EC0C20
sokol@server:~/test$ ./getrand
F31CE56C


sokol@server:~/test$ cat getrand
#!/bin/bash
target=google.com
#c=0
#while [ $c -lt 8 ]
#do
#c=$[$c+1]

{
echo "obase = 16"
ping -c 8 -i 0.2 $target |\
grep --line-buffered "time=" |\
sed -u \
-e 's/[a-z \.(),_:+=-]//g' \
-e 's/\(.\)/\1+/g' \
-e 's/^\(.*\)+$/(\1)% 16/'
} |\
bc |\
tr -d '\n'

echo ""
#done

In the code commented out is the ability to run this in a loop, also if your not going to enjoy the output in real-time you can remove the --line-buffered from grep and -u from sed, it may make a small performance improvement    Keep in mind each 4 bit hex work is the result of one ping. So this is taking up network resources.

Generation of 8x8 tile Random Hex Digits

sokol@server:~/test$ sudo ./getrand2
7BDBF0AA
6BC6BC42
B63F2397
D55A5A5C
F10A1A7D
112329BC
110099F7
C1F53C98

sokol@server:~/test$ cat getrand2
#!/bin/bash
target=google.com
c=0
while [ $c -lt 8 ]
do
c=$[$c+1]

{
echo "obase = 16"
ping -c 8 -i 0.02 $target |\
grep "time=" |\
sed \
-e 's/[a-z \.(),_:+=-]//g' \
-e 's/\(.\)/\1+/g' \
-e 's/^\(.*\)+$/(\1)% 16/'
} |\
bc |\
tr -d '\n'
echo ""
done

This is easy enough to adapt for any purpose, and even redirect output to a file.

you need to be root to set the ping Interval time to below 0.2. Sudo does this.

Under the Hood


Below I peel off each layer to show how it's being done.


This is what just ping generates

sokol@server:~/test$ sudo ping -c 8 -i 0.02 google.com | grep "time="
64 bytes from 173.194.64.99: icmp_seq=1 ttl=47 time=8.86 ms
64 bytes from 173.194.64.99: icmp_seq=2 ttl=47 time=8.87 ms
64 bytes from 173.194.64.99: icmp_seq=3 ttl=47 time=8.86 ms
64 bytes from 173.194.64.99: icmp_seq=4 ttl=47 time=8.08 ms
64 bytes from 173.194.64.99: icmp_seq=5 ttl=47 time=8.08 ms
64 bytes from 173.194.64.99: icmp_seq=6 ttl=47 time=8.85 ms
64 bytes from 173.194.64.99: icmp_seq=7 ttl=47 time=8.97 ms
64 bytes from 173.194.64.99: icmp_seq=8 ttl=47 time=8.93 ms


After everything but the number have been stripped out, added together and modulo 16.

sokol@server:~/test$ sudo ping -c 8 -i 0.02 google.com | grep "time=" | sed -e 's/[a-z \.(),_:+=-]//g' -e 's/\(.\)/\1+/g' -e 's/^\(.*\)+$/(\1)% 16/'
(6+4+1+7+3+1+9+4+6+4+9+9+1+4+7+8+9+4)% 16
(6+4+1+7+3+1+9+4+6+4+9+9+2+4+7+8+9+2)% 16
(6+4+1+7+3+1+9+4+6+4+9+9+3+4+7+8+9+1)% 16
(6+4+1+7+3+1+9+4+6+4+9+9+4+4+7+7+9+8)% 16
(6+4+1+7+3+1+9+4+6+4+9+9+5+4+7+7+9+8)% 16
(6+4+1+7+3+1+9+4+6+4+9+9+6+4+7+7+9+8)% 16
(6+4+1+7+3+1+9+4+6+4+9+9+7+4+7+7+9+6)% 16
(6+4+1+7+3+1+9+4+6+4+9+9+8+4+7+8+9+3)% 16

What bc generates for it's output. But I send it the command "obase=16" first that makes it's output Hex.

sokol@server:~/test$ sudo ping -c 8 -i 0.02 google.com | grep "time=" | sed -e 's/[a-z \.(),_:+=-]//g' -e 's/\(.\)/\1+/g' -e 's/^\(.*\)+$/(\1)% 16/' | bc
5
4
6
9
6
7
13
10


Read also: Testing ping based Random number generator

Sunday, October 30, 2011

CGI's in BASH: part 1

It's possible to write quick and dirty cgi scripts in Bash that quickly report information about a system.

uptime

#!/bin/bash
echo Content-type: text/plain
echo
/usr/bin/uptime

Or you can expand this a little

#!/bin/bash
UPTIME=/usr/bin/uptime


echo Content-type: text/plain
echo
if [ -x $UPTIME ];  then
        $UPTIME
else
        echo Cannot find uptime command on this system.
fi

CGI's in BASH, Part 2: Conclusion of Domain report

This is the conclusion to
Checking Status of Domains hosted on a server. Advanced Bash

This CGI in bash take the data we harvested in check2 from the out directory and prints this output to the web in a clean format. 

It take the list of domains from "masterlist" and searches the output files to see if that domain is in the good or bad bucket. 

It then add the results to an HTML table being constructed. 


domainreport
#!/bin/bash

echo "Content-type: text/html"
echo
echo "<html><body>"


cd /data/namedb/checkdomains

echo "<table border=2 cellpadding=2 cellspacing=2>"

for h in $( cat masterlist ); do

out="<tr><td>$h </td> "

for i in "extern" "local" "sokol" "whois" ; do

out="$out <td>`grep -l "^$h" $i.*`</td>"

done

out="$out <td>`grep -l "^$h" domains-from-httpd`</td>"
out="$out <td>`grep -l "^$h" domains-from-mail`</td>"
out="$out <td>`grep -l "^$h" available`</td>"
out="$out </tr>"

echo $out |\
sed -e 's/domains-from-httpd/Httpd/' |\
sed -e 's/domains-from-mail/E-Mail/' |\
sed -e 's/\.bad/<font color="red">\0<\/font>/g' |\
sed -e 's/\.good/<font color="green">\0<\/font>/g'
done

echo "</table>"
echo "</body></html>"


This second version save the first pass output to a file, then takes a second pass to sort the report before removing the index used to sort, and then displays it.


domainreport2
#!/bin/bash

echo "Content-type: text/html"
echo
echo "<html><body>"

echo "" > /tmp/domainreportout

cd /data/namedb/checkdomains

echo "<table border=1 cellpadding=2 cellspacing=2>"

for h in $( cat masterlist ); do

out="<td><a target="blank" href="http://$h">$h</a></td>"

for i in "extern" "local" "sokol" "whois" ; do

out="$out <td>`grep -l "^$h" $i.*`</td>"

done

out="$out <td>`grep -l "^$h" domains-from-httpd`</td>"
out="$out <td>`grep -l "^$h" domains-from-mail`</td>"
out="$out <td>`grep -l "^$h" available`</td>"
tally=`echo $out | tr '<' '\n' | grep -c "\.good"`
#out="<td>$tally</td>$out"
out="$tally ::$out"
out="<tr>$out</tr>"

echo $out |\
sed -e 's/domains-from-httpd/Httpd/' |\
sed -e 's/domains-from-mail/E-Mail/' |\
sed -e 's/\.bad/<font color="red">\0<\/font>/g' |\
sed -e 's/\.good/<font color="green">\0<\/font>/g' >> /tmp/domainreportout
done

sort -n -r /tmp/domainreportout | sed 's/.*:://'


echo "</table>"
echo "</body></html>"


Below is an example of the output.


6ghz.comextern.goodlocal.goodsokol.goodwhois.goodHttpdE-Mail
8vsb.comextern.goodlocal.goodsokol.goodwhois.goodHttpdE-Mail
9bnog.comextern.badlocal.badsokol.goodwhois.bad
9bnog.netextern.badlocal.badsokol.goodwhois.bad
9bnog.orgextern.badlocal.badsokol.goodwhois.bad
a2znetworks.netsokol.badwhois.badE-Mailavailable

Checking Status of Domains hosted on a server. Advanced Bash

This is a 2 pass system to report of domain status of domains hosted on a server.

I have a domains file I use to automatically generate DNS entries.
   Part 2 here, output reports to the web


PASS1 - Harvest list of domains, and gather whois records and DNS lookups.
#!/bin/bash

# Harvest Domains from QMAIL
sort -u /var/qmail/control/locals /var/qmail/control/plusdomain /var/qmail/control/rcpthosts > domains-from-mail


# Harvest Domains from Apache Config
grep -E "ServerName|ServerAlias" /conf/sites-enabled/*  |\
 awk '{print $NF}' |\
 awk -F "." '{print $(-1 + NF)"."$NF}' |\
 sort -u > domains-from-httpd

# Copy over other sources of Domains on the host,  may want to harvest DNS records too.


# Consolidate all domains know to the system from all domains* files in to masterlist of domains to perform checks.
cat domains* | awk '{ print $NF }' | sort -u > masterlist

# This version supports comments, change out with line above to do that.
#cat domains* | grep -v "^#" | awk '{ print $NF }' | sort -u > masterlist

# Make sure there is an out directory

# clean out bad entries in out Directory
grep -l "LIMIT EXCEEDED" out/*.whois | xargs rm


# Harvest data from local and external DNS & whois
for h in $( cat masterlist ); do

host $h > out/$h.local &
host $h 8.8.8.8  > out/$h.extern &

#ls -l out/$h.whois
if [ ! -s out/$h.whois ]; then
#echo "Running whois on out/$h.whois"
whois $h > out/$h.whois &
fi
done

sleep 2
grep -l "LIMIT EXCEEDED" out/*.whois | xargs rm


For each domain we generate 3 files, the external DNS lookup, the Internal lookup and the whois lookup.

-rw-r--r-- 1 sokol sokol 148 Oct 28 20:32 6ghz.com.extern
-rw-r--r-- 1 sokol sokol 82 Oct 28 20:32 6ghz.com.local
-rw-r--r-- 1 sokol sokol 4388 Oct 28 20:08 6ghz.com.whois
-rw-r--r-- 1 sokol sokol 148 Oct 28 20:32 8vsb.com.extern
-rw-r--r-- 1 sokol sokol 82 Oct 28 20:32 8vsb.com.local
-rw-r--r-- 1 sokol sokol 4388 Oct 28 20:08 8vsb.com.whois

Before we generate a report we need to harvest the useful information from the data we just gathered.

This is done with simple grep searches and depositing the output in to a good or bad bucket.


PASS2 - this organized the data we just harvested.
#!/bin/bash

MYDNS="72.249.144.147"
MYHOST="DNULL.COM"
MYNAME="sokol"

grep address out/*.extern | grep -v $MYDNS | sed -e 's/.*://' > extern.bad
grep address out/*.extern | grep $MYDNS | sed -e 's/.*://' | awk '{print $1}' > extern.good

grep address out/*.local | grep -v $MYDNS | sed -e 's/.*://' > local.bad
grep address out/*.local | grep $MYDNS | sed -e 's/.*://' | awk '{print $1}' > local.good

grep -L -i $MYHOST out/*.whois | sed -e 's/\.whois//' -e 's/.*\///' > whois.bad
grep -l -i $MYHOST out/*.whois | sed -e 's/\.whois//' -e 's/.*\///' > whois.good

grep -L -i $MYNAME out/*.whois | sed -e 's/\.whois//' -e 's/.*\///' > sokol.bad
grep -l -i $MYNAME out/*.whois | sed -e 's/\.whois//' -e 's/.*\///' > sokol.good

grep -l "No match for" out/*.whois | sed -e 's/\.whois//' -e 's/.*\///' > available


REPORT
#!/bin/bash
for h in $( cat masterlist ); do

out="$h: "

for i in "extern" "local" "sokol" "whois" ; do
out="$out `grep -l "^$h" $i.*`"
done

out="$out `grep -l "^$h" domains-from-httpd`"

echo $out |\
sed -e 's/domains-from-httpd/httpd/' |\
sed -e 's/\.bad/\d27[32;41m\0\d27[m/g' |\
sed -e 's/\.good/\d27[30;42m\0\d27[m/g'
done
Read the next article to see this in CGI form.

Tuesday, October 25, 2011

Bash & shell trick

This is a rehash of some old notes I had on the wiki.


Wiki tricks
Move heading in a level deeper
sed -e 's/\(=\+\)/=\1/g' infile > outfile

Wikipedia:Tools/Editing_tools
An HTML to Wiki syntax converter
Convert Word doc or Webpage to wiki
http://excel2wiki.net/

Sed tricks



Bash Tricks


RPM


Harvest host memory profile

gethostinfo.sh
/usr/sbin/dmidecode > /tmp/DMIDECODE
LLIST=`grep -n "Memory Device" /tmp/DMIDECODE | grep -v "Address" | sed 's/:.*//'`
TOTSIZE=0
for i in $LLIST; do
MEMSIZE=`tail -n +$i /tmp/DMIDECODE | head -13 | grep "Size:" | sed 's/No/0/' | awk '{print $2}'`

  TOTSIZE=$[ $TOTSIZE + $MEMSIZE ]
done
PNAME=`grep "Product Name" /tmp/DMIDECODE |  sed  -e 's/.*: //' -e 's/ /_/g' -e '1 q' `
HNAME=`hostname | sed 's/\.xyzcompany.*//'`

OSINFO=`uname -n -m -r`
#SPEEDINFO=`grep "Current Speed:" /tmp/DMIDECODE | sed  -e 's/.*: //' -e 's/000 MHz/ GHz/' -e '1 q' `
NUMCORES=`grep "^processor" /proc/cpuinfo |  sed  -e 's/.*: //'   | tail -1`

NUMCORES=$[ $NUMCORES + 1]
CPUINFO=`grep "model name" /proc/cpuinfo |  sed  -e 's/.*: *//' -e 's/\s\+/_/g' -e '1 q' `
RAMSIZE=`echo "$TOTSIZE MB" | sed -e 's/1024 MB/1 GB/' -e 's/2048 MB/2 GB/' -e 's/4096 MB/4 GB/' -e 's/6144 MB/6 GB/' -e 's/8192 MB/8 GB/' -e 's/12288 MB/12 GB/' -e 's/16384 MB/16 GB/' `

#echo "$OSINFO $PNAME $RAMSIZE $SPEEDINFO"
echo "$OSINFO, $PNAME, $RAMSIZE, $NUMCORES Core, $CPUINFO"



TCPDUMP

sudo tcpdump -i bond0:6 -s 0 -x host 10.1.224.147 and port 8080
sudo tcpdump -i bond0:6 -s 0
sudo tcpdump -i bond0:6 -s 0 -X port 8080 -w /tmp/tdump1
output file tdump1 can be read in using WireShark

rsync

Pulls files off remote and delete's then after verified transfer.
sudo rsync -auvlz  --remove-sent-files jsokol@xyzhost:tmp/xyx* .



search and kill processes
This was for finding and killing XYZ processes on servers, (it's assumed) they will restart themselves.

[jsokol]$ more bouncehosts
if [ -s $1 ] ; then
 awk '{ print $1 }' $1 > /tmp/hostlist1

        for h in $(cat /tmp/hostlist1 );do
        echo ID $h
TMP1=`ps ax | grep xyz | grep Builds | grep "com_$h "`

#         ps ax | grep xyz | grep Builds | grep "com_$h "
        if [ "$?" -eq 1 ] ; then

            TMP2=`grep "^$h " $1`
            echo No XYZ Running for $TMP2
        else

            TMP2=`echo $TMP1 | awk '{print $1}'`
            echo Killing Pid $TMP2
            kill $TMP2
            sleep 30
        fi

#        grep "$h " x2 | awk '{print "'$h' " $1}' | sed 's/XYZ=/ /' | sed 's/%2C/  /g' | sed 's/.xyzco.com/ /g'


        done

else
        echo "file $1 doesn't exist, or you didn't enter a filename"
fi

if [ "$UID" -ne "0" ];then echo "please sUd0 this.."; exit
ssh -q "$3"@hostdb.corp "sudo mysql -e \"select hostname,productname from host where productname like '%$1' and hostname like '%$2%' \" noc";
for i in $(cat XYZ.all); do echo "edit" | ssh sokol@$i 'sudo sed s/34/37/g -i /etc/app/configfile' ;done >> xyz.log

[jsokol@xyz.sjc ~]$ ZZ=6
[jsokol@xyz.sjc ~]$ echo $[4 + $ZZ]
10

echo xyzhost(3..6}.sjc,|sed s/", "/","/g
xyzhost3.sjc,xyzhost4.sjc,xyzhost5.sjc,xyzhost6.sjc,


running average

#!/bin/bash
# example of how to get running average.
if [ ! -s "runaveragedata-last" ] ; then
  echo 0 > runaveragedata-last
fi

NEWVAL=`cat newdata`
RUNAVR=`cat runaveragedata-last`

# doing work in fixpoint numbers  
NEWAVR=$[ $NEWVAL * 1000 + $[ $RUNAVR * 9 ] ]
RUNAVR=$[ $NEWAVR / 10 ]
echo $RUNAVR > runaveragedata-last
echo $[ RUNAVR / 1000 ]

Bash loops

for h in 07 08 09 10 11 12 13 18; do scp host:/path /xyc$h.log .;done

for h in $(cat file ); do echo $h |tee -a outputlog; rsync -auvlz host:$h $h; done

rpm -qa | egrep -i 'egw|logfs' | xargs sudo rpm -e


------------
more dupesniff.sh
#!/bin/bash

for h in $(grep -vE ^# $1  | grep -vE ^$ | awk {'print $1'} | sort | uniq -d )
do
        grep -n $h $1
done

------------

[jsokol]$ cat portprobe.pl

#!/usr/bin/perl -w
#
#  tiny tool meant to mimic a device profile.  It should take ports, or service names,
#  and mimic the open ports epected for that type of machine
#

use IO::Socket;
use Sys::Hostname;


my $proto = "udp";
my $port = "8080";
my $hostname = hostname;
my $persist = 0;
my $server = "localhost";

my $socket = IO::Socket::INET->new(PeerAddr     => $server,

                                PeerPort        => $port,
                                Proto           => $proto,
                                Type            => SOCK_STREAM)
                        or die "Couldn't talk to $server on $port \n\n";




print $socket "GET  \n";
@answer = <$socket>;
close($socket);
print "@answer \n\n";

------------
[jsokol]$ cat portlisten.pl

#!/usr/bin/perl -w
#
#  tiny tool meant to mimic a device profile.  It should take ports, or service names,
#  and mimic the open ports epected for that type of machine
#
#

use IO::Socket;
use Sys::Hostname;


my $proto = "udp";
my $port = "8080";
my $hostname = hostname;
my $persist = 0;

my $server = IO::Socket::INET->new(     LocalPort       => $port,

                #                       Type            => SOCK_STREAM,
                                        Proto           => $proto,
                #                       Reuse           => 1,
                #                       Listen          => 10

                                        ) or die
                "\tCouldn't generate new server socket on port $port\n\n";

print "starting server\n\n";
$server->accept();
while ($server->recv($msg, 255)) {

        print "Got message $msg from $server->peername \n";
        $server->send("Thanks");
}

close($server);
------------

#!/bin/sh

# comm-hack.  Use this to send multiple commands to multiple hosts.  
# Usage: ./scriptname [hostlist filename] [user]


x=$1 #host list
y=$2 #username

#begin

echo
echo -n "Enter password: "
read -s PASS
echo
echo -n "Enter commands: "

read COMMANDS

#for i in $(cat $x ); do echo -e "$PASS" | ssh '$y'@$i $COMMANDS; done

for i in $(cat $x ); do echo "$PASS"| ssh "$y"@$i $COMMANDS;
echo "Executed on host: $i @ `date '+%m/%d/%y%t'` "

done
fi

Friday, January 07, 2011

Bash: Find the Max in a list of Integers

Max is a little shells script that will find the maximum value and returns that value and the line it was found at.

max
#!/bin/bash
MAX=0
ELE=0
while read line; do
ELE=$[ $ELE + 1 ]
if [ "$line" -gt "$MAX" ]; then
MAX=$line
POS=$ELE
fi
done
echo "$MAX $POS"

Usage Example
-bash-3.2$ cat ttt
1000000001
1000000000002
1000000000007
1000000000033
9999
1000000000000
1000000000005
-bash-3.2$ ./max < ttt
1000000000033 4
-bash-3.2$

Tuesday, December 21, 2010

The Mysteries of the Unix Date Command.

To do any math on dates is very difficult in any language. But in the Unix/Linux shell this can be a breeze.

This is done by converting all data time in to Seconds.

In Unix this is done by counting seconds from UTC or Universal Constant Time, 1/1/1970

So once you convert in to second, you can do math as usual, then convert time back to date time.


Example of how to convert back and forth

Conversion from Seconds UTC to string
-bash-3.2$ date --date "1970-01-01 1292970890 sec utc"
Tue Dec 21 14:34:50 PST 2010

Conversion from string to Seconds UTC
-bash-3.2$ date --date "12/21/2010 14:34:50 PST" "+%s"
1292970890
-bash-3.2$ date --date "Dec 21 14:34:50 PST 2010" "+%s"
1292970890

Current time
-bash-3.2$ date "+%s"
1292979451

Get Seconds UTC file creation date.
stat -c %Y filename
ls -l
-rw-r--r-- 1 jsokol jsokol       8 Sep 22 18:36 t
 
-bash-3.2$ stat -c %Y t
1285205797


So for doing math on time:


Get file age

(In Bash)
Returns how many hours old
expr \( `date +%s` - `stat -c %Z $filename` \) / 3600

Return age of oldest file in directory
expr \( `date +%s` - `stat -c %Y \`ls -t | tail -1\` ` \) / 3600

Create Shell alias of this
alias oldest='expr \( `date +%s` - `stat -c %Y \`ls -t | tail -1\` ` \) / 3600'