One-liner: how to count the total amount of specific lines in several files

Let’s calculate the amount of lines, containing the word “extended” in */*.conf files:

egrep -c extended */*.conf | 
awk 'BEGIN {FS=":"; sum=0;}{sum +=$2} END {print sum}'  

BTW, using awk alone is slower:

time awk 'BEGIN {sum=0;}/extended/{sum++} END {print sum}' */*.conf
110653

real	0m0.94s
user	0m0.91s
sys	0m0.01s

time egrep -c extended */*.conf | 
awk 'BEGIN {FS=":"; sum=0;}{sum +=$2} END {print sum}'
110653

real	0m0.13s
user	0m0.10s
sys	0m0.02s

fgrep is slower than egrep:

time fgrep -c extended */*.conf | 
awk 'BEGIN {FS=":"; sum=0;}{sum +=$2} END {print sum}'
110653

real	0m0.21s
user	0m0.17s
sys	0m0.03s

traceroute 216.81.59.173

> traceroute -m 100 216.81.59.173
traceroute: Warning: Multiple interfaces found; using x.x.x.x @ net0
traceroute to 216.81.59.173 (216.81.59.173), 30 hops max, 40 byte packets
. . .
8  10gigabitethernet1-2.core1.atl1.he.net (184.105.213.110)  122.807 ms  150.309 ms  168.517 ms
9  216.66.0.26 (216.66.0.26)  160.820 ms  164.675 ms  157.556 ms
10  * * *
11  Episode.IV (206.214.251.1)  188.004 ms  188.078 ms  277.575 ms
12  A.NEW.HOPE (206.214.251.6)  212.980 ms  182.796 ms  217.315 ms
13  It.is.a.period.of.civil.war (206.214.251.9)  208.230 ms  231.501 ms  187.249 ms
14  Rebel.spaceships (206.214.251.14)  223.330 ms  185.769 ms  231.825 ms
15  striking.from.a.hidden.base (206.214.251.17)  222.702 ms  199.810 ms  227.345 ms
16  have.won.their.first.victory (206.214.251.22)  186.517 ms  221.058 ms  201.745 ms
17  against.the.evil.Galactic.Empire (206.214.251.25)  185.988 ms  216.445 ms  186.553 ms
 Continue reading "traceroute 216.81.59.173"

Simple log-rotator

Non-compressing log-rotator:

# more logrotate.sh
#!/bin/bash
for file in $*
do
mv ${file}.5 ${file}.6
mv ${file}.4 ${file}.5
mv ${file}.3 ${file}.4
mv ${file}.2 ${file}.3
mv ${file}.1 ${file}.2
mv ${file} ${file}.1
touch ${file}
pkill -1 syslog
done

Compressing version:

#!/bin/bash
for file in $*
do
mv ${file}.5.gz ${file}.6.gz
mv ${file}.4.gz ${file}.5.gz
mv ${file}.3.gz ${file}.4.gz
mv ${file}.2.gz ${file}.3.gz
mv ${file}.1.gz ${file}.2.gz
mv ${file} ${file}.1
touch ${file}
pkill -1 syslog
gzip ${file}.1
done

Usage:

# logrotate.sh asa1.log cisco2.log templog
#

One-liners: simple ASA log parsers

Sometimes you do not need a detailed log-analysis but several simple one-liners that you can adjust without too much thinking how it works, what you did last time, etc. The examples below are absolutely NOT optimal, but rather modular for easy line-editing.

1. Allowed traffic:

cat   asa.log | grep permitted | grep access-list |
sed -e 's/^.*permitted//' -e 's/hit-cnt.*$//' |
sed -e 's/([0-9][0-9]*) ->/ ->/' | less

Result:

 tcp outside/10.2.8.30 -> inside/10.1.141.23(8080)
 tcp outside/10.2.8.30 -> inside/10.1.141.23(8080)
 tcp outside/10.2.8.35 -> inside/10.1.140.137(13000)
 udp outside/10.2.8.25 -> inside/10.1.9.14(137)
 udp outside/10.2.8.25 -> inside/10.1.81.15(137)
 tcp outside/10.2.8.44 -> inside/10.1.140.137(13000)
 tcp outside/10.2.8.31 -> inside/10.1.140.149(13000)

Noticed the “permitted” strings?

2. Denied traffic:

cat   asa.log | grep denied | grep access-list |
sed -e 's/^.*denied//' -e 's/hit-cnt.*$//' |
sed -e 's/([0-9][0-9]*) ->/ ->/' | less

Result:

 tcp inside/10.1.140.159 -> outside/10.2.8.24(515)
 tcp inside/10.1.140.159 -> outside/10.2.8.24(515)
 tcp inside/10.3.241.116 -> outside/10.2.8.251(1541)
 tcp inside/10.3.241.116 -> outside/10.2.8.251(1547)
 tcp inside/10.1.140.159 -> outside/10.2.8.24(515)

3. The most popular permitted traffic:

cat   asa.log | grep permitted | grep access-list | 
sed -e 's/^.*permitted//' -e 's/hit-cnt.*$//' |
sed -e 's/([0-9][0-9]*) ->/ ->/' |
awk ' {conn[$0]++;} END { for ( i in conn ) print conn[i],"	",i;}' | sort +0nr | less

Result (the first column is the amount of corresponding log entries):

21170   tcp outside/10.2.8.40 -> inside/10.1.140.149(13000)
18023   tcp outside/10.2.8.34 -> inside/10.1.140.149(13000)
17981   tcp outside/10.2.8.31 -> inside/10.1.140.149(13000)
11034   tcp inside/10.1.140.251 -> outside/10.2.8.68(10001)
10652   tcp outside/10.2.8.43 -> inside/10.1.140.137(13000)
10628   tcp outside/10.2.8.44 -> inside/10.1.140.137(13000)
10484   tcp outside/10.2.8.47 -> inside/10.1.140.137(13000)
10437   tcp outside/10.2.8.23 -> inside/10.1.140.137(13000)
7618   tcp outside/10.2.8.25 -> inside/10.1.140.137(13000)
7550   tcp outside/10.2.8.27 -> inside/10.1.140.137(13000)
7515   tcp outside/10.2.8.49 -> inside/10.1.140.137(13000)
7496   tcp outside/10.2.8.29 -> inside/10.1.140.137(13000)
6826   tcp outside/10.2.8.30 -> inside/10.1.141.23(8080)
6011   tcp outside/10.2.8.35 -> inside/10.1.140.137(13000)
5896   tcp outside/10.2.8.40 -> inside/10.1.141.23(8080)
5809   tcp outside/10.2.8.30 -> inside/10.1.140.137(13000)

4. Modification for versions 8.2+:

cat asa.log | grep Deny | grep access-group | sed -e 's/^.*Deny//' -e 's/by.*$//' | sed -e 's/dst/ ->/' -e 's/src//' | less

You can use this method to generate a policy based on the current traffic.

Additional “greps” will allow you to filter for specific ports or IP-addresses.

Replace “permitted” with “denied” and you’ll get the “most popular denied traffic”.

What directory PHP is busy with

The file “cwd” under /proc/pid is a symbolic link to the “current working directory”:

for i in `ps -ef | awk '/php/{print $2}'`
do
ls -l /proc/${i}/cwd
done

The result is:

lrwxrwxrwx 1 alekz alekz 0 Янв 25 02:40 /proc/11544/cwd -> /home/alekz/public_html/blog/wp
lrwxrwxrwx 1 alekz alekz 0 Янв 25 02:40 /proc/11764/cwd -> /home/alekz/public_html/alekz.net
lrwxrwxrwx 1 alekz alekz 0 Янв 25 02:40 /proc/12574/cwd -> /home/alekz/public_html/alekz.net
lrwxrwxrwx 1 alekz alekz 0 Янв 25 02:40 /proc/13081/cwd -> /home/alekz/public_html/alekz.net
lrwxrwxrwx 1 alekz alekz 0 Янв 25 02:45 /proc/15053/cwd -> /home/alekz/public_html/blog/wp
lrwxrwxrwx 1 alekz alekz 0 Янв 25 02:45 /proc/15056/cwd -> /home/alekz/public_html/blog/wp
lrwxrwxrwx 1 alekz alekz 0 Янв 25 02:49 /proc/15696/cwd -> /home/alekz/public_html/blog/wp/wp-content/plugins/si-captcha-for-wordpress/captcha-secureimage

Of course, you can use any other process name instead of php.

And here’s a “dynamic” version:

while true
do
clear
for i in `ps -ef | awk '/php/{print $2}'`
do
ls -l /proc/${i}/cwd
done
sleep 5
done

dcpumon and dcpumonview

Found another cPanel application which you can safely turn off in order to safe some bytes and cycles.

By default dcpumon runs every 5 min to log CPU usage (“top” output) :

# crontab -l | fgrep cpu
*/5 * * * * /usr/local/cpanel/bin/dcpumon >/dev/null 2>&1
#

and stores the data into /var/log/dcpumon

You can view the report with dcpumonview command:

# /usr/local/cpanel/bin/dcpumonview
———————————————————–
|User |Domain |CPU%|MEM%|MySQL#|
———————————————————–
|alekz |alekz.net |17.72|37.07|0.3 |
| Top Process | 27.8 | /usr/bin/php |
| Top Process | 14.2 | /usr/bin/php |
| Top Process | 12.3 | /usr/bin/php |
|mysql | |11.47|3.05|0.0 |
| Top Process | 11.7 | /usr/sbin/mysqld –basedir/ –datadir/var/lib/mysql –usermysql –pid-file/var/lib/mysql/alekz.pid –skip-external-locking |

QNAP NAS, Solaris and NFS

I thought that nothing could be easier on Solaris than to mount a directory from a QNAP NAS (running Linux). Ha! Nope… Here’s some gotchas.

QNAP firmware v2.x does not support NFSv3 or NFSv4, so you have to explicitly use “-o vers=2”. As a result, you cannot use files bigger than 2GB (no problems with SMB).

QNAP firmware v3.2.x supports NFSv3 (use “-o vers=3”) but does not support NFSv4. Big files are back, but to get the AUTH_SYS authentication back too, add “sec=sys” to the options string (considering that users on the NAS box have the same UID’s as on Solaris).

xeon# mount -F nfs nas:/photo /mnt
nfs mount: nas:/photo: No such file or directoryxeon# mount -F nfs -o vers=3,sec=sys nas:/photo /mnt
xeon#

Or use the automounter:

xeon# more /etc/auto_master
. . .
/nas auto_nas -nobrowsexeon# more /etc/auto_nas
* -vers=3,sec=sys nas:/&

xeon# svcadm restart autofs
xeon#

xeon# cd /nas/video
xeon#

The directories must be exported as “No Limit” access rights (UID’s and standard file access modes will be used).