UNIX Archives - AlekZ' Scratchpad

One-liner: How to scan a network for up & down hosts and show OUI info

This is another version of the ping sweep one-liner script. To show the OUI (Ethernet interface manufacturer) it needs to be run locally (to scan the locally connected networks).

Ubuntu version:

for i in {1..10}
 do 
  ( ip="10.0.1.$i"
  up=`ping -n -q -W1 -c2 $ip | fgrep transmitted | sed -e 's/.*100%.*/down/' -e 's/.*0%.*/up/'`
  mac=`arp -an $ip | awk '{print $4}'`
  oui=`fgrep $(echo $mac | tr -d ":" | head -c6 | tr '[a-z]' '[A-Z]') /usr/share/ieee-data/oui.txt | sed -e 's/^.*)..//'`
  name=`host $ip | awk '/pointer/{print $NF}'`
  echo $ip $up $name $mac $oui ) 2>/dev/null &  
done | sed -e 's/<incomplete>//'  -e 's/entries//'  | sort -n -t. -k4

Result:

10.0.1.1 down 
10.0.1.2 down 
10.0.1.3 up host1. 00:08:9b:c0:ff:ee ICP Electronics Inc.
10.0.1.4 down 
10.0.1.5 down host2. 
10.0.1.6 up nas3. 52:54:00:c0:ff:ee
10.0.1.7 down laptop. 
10.0.1.8 down 
10.0.1.9 up switch2. 88:51:fb:c0:ff:ee Hewlett Packard
10.0.1.10 down printer.

RedHat/CheckPoint firewall version:

for i in {1..10}
 do 
  (ip="10.0.1.$i"
  up=`ping -n -q -W1 -c2 $ip | fgrep transmitted | sed -e 's/.*100%.*/down/' -e 's/.*0%.*/up/'`
  mac=`arp -an $ip | awk '{print $4}'`
  oui=`fgrep $(echo $mac | tr -d ":" | head -c6 | tr '[a-z]' '[A-Z]')  /usr/share/hwdata/oui.txt | sed -e 's/^.*)..//'`
  name=`host $ip | awk '/pointer/{print $NF}'`
  echo $ip $up $name $mac $oui ) 2>/dev/null &  
done | sed -e 's/<incomplete>//' -e 's/entries//'  | sort -n -t. -k4

The OUI database /usr/share/ieee-data/oui.txt (for Ubuntu) or /usr/share/hwdata/oui.txt (for RedHat/CheckPoint) should be kept up-to-date.

Semi-colon separated version for Ubuntu:

for i in {1..10}
 do 
  (ip="10.0.1.$i"
  up=`ping -n -q -W1 -c2 $ip | fgrep transmitted | sed -e 's/.*100%.*/down/' -e 's/.*0%.*/up/'`
  mac=`arp -an $ip | awk '{print $4}'`
  oui=`fgrep $(echo $mac | tr -d ":" | head -c6 | tr '[a-z]' '[A-Z]') /usr/share/ieee-data/oui.txt | sed -e 's/^.*)..//'`
  name=`host $ip | awk '/pointer/{print $NF}'`
  echo "$ip;$up;$name;$mac;$oui"  ) 2>/dev/null &  
done | sed -e 's/<incomplete>//'  -e 's/entries//'  | sort -n -t. -k4

Result:

10.0.1.1;down;;;
10.0.1.2;down;;;
10.0.1.3;up;host1.;00:08:9b:c0:ff:ee;ICP Electronics Inc.
10.0.1.4;down;;;
10.0.1.5;down;host2.;;
10.0.1.6;up;nas3.;52:54:00:c0:ff:ee;
10.0.1.7;down;laptop.;;
10.0.1.8;down;;;
10.0.1.9;up;switch2.;88:51:fb:c0:ff:ee;Hewlett Packard
10.0.1.10;down;printer.;;

One-liner: How to scan a network for up & down hosts

Another network scanner.

For example, you want to find possible free IP-addresses in the 10.0.10.1 – 10.0.10.1 range:

for i in {1..10}; do (ip="10.0.10.$i"; up=`ping -n -q -W3 -c1 $ip | fgrep transmitted | sed -e 's/.*100%.*/down/' -e 's/.*0%.*/up/'`;name=`host $ip | awk '!/NXDOMAIN/{print $NF}'`; echo $ip $up $name)  2>/dev/null & done | sort -n -t. -k4

Result:

10.0.10.1 down
10.0.10.2 down
10.0.10.3 up host1.
10.0.10.4 down
10.0.10.5 down myhost2.internal.domain.
10.0.10.6 up nas3.
10.0.10.7 down laptop.
10.0.10.8 down
10.0.10.9 up switch2.
10.0.10.10 down printer.

One-liner: how to change file extensions according to their type

What if you have PNG or WEBP files, saved with .jpg extension? Here’s a one liner to rename them accordingly. You need mediainfo package:

sudo apt-get install mediainfo

Imagine we have file1.jpg in the current directory:

% ls
file1.jpg
% file file1.jpg
file1.jpg: RIFF (little-endian) data, Web/P image, VP8 encoding, 800x800, Scaling: [none]x[none], YUV color, decoders should clamp

No problem just rename it! But what if we have a hundred?

for i in * 
do 
  oldext=`echo "$i" | sed -e 's/^.*\.\([A-Za-z][A-Za-z]*\)$/\1/'` 
  echo "Old ext=$oldext" 
  newext=`mediainfo "$i" | egrep "^Format[[:space:]]+:" | sort -u | tr '[A-Z]' '[a-z]' | sed -e 's/.* \([a-z][a-z]*\)/\1/' -e 's/jpeg/jpg/'` 
  echo "New ext=$newext" 

  oldname=`echo $i | sed -e 's/\.'$oldext'$/.'$oldext'/'` 
  newname=`echo $i | sed -e 's/\.'$oldext'$/.'$newext'/'` 

  echo old=$oldname 
  new=$newname 
  mv -i $oldname $newname 

done

The result:

Old ext=jpg
New ext=webp
old=file1.jpg new=file1.webp

% ls
file1.webp

One-liner: How to remove spaces in file names

Spaces in file names are nasty and might prevent various script from working properly. Here’s a one-liner to replace spaces with underscore for all files in the current directory:

for i in *
do
   mv "$i" ${i// /_}
done

One-liner: How to check positional numbers of elements in the CheckPoint logs

CheckPoint log entries are divided by semi-colons and can have … many … fields. How to quickly check the positional number of a particular field in a particular log entry? Here’s a quick AWK one-liner (in AWK the “0” element is the whole line):

$ echo '315918;1Jan2019;0:03:30;fe80::d123:3aaa:fe80:fb73;ff02::1;ipv6-icmp;;accept;;;;10.1.2.26;log;;eth1.123;inbound;VPN-1 & FireWall-1;;f-firewall001;Network;0;;;;;;;;;Implied rule;;;Neighbor Advertisement;136;0;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;' | \ 
awk -v RS=\; '{print NR,$0}'
1 315918
2 1Jan2019
3 0:03:30
4 fe80::d123:3aaa:fe80:fb73
5 ff02::1
6 ipv6-icmp
7 
8 accept
9 
10 
11 
12 10.1.2.26
13 log
14 
15 eth1.123
16 inbound
17 VPN-1 & FireWall-1
18 
19 f-firewall001
20 Network
21 0
22 
23 
24 
25 
26 
27 
28 
29 
30 Implied rule
31 
32 
33 Neighbor Advertisement
34 136
35 0
36 
37 
...
92

A bit longer alternative variant:

awk -F\; '{ for (i=1;i<=NF;i++) {print i,$i}}'

To number the field names (provided the logs are converted to TXT and gzipped):

$ zcat 2019-01-01_025249_2308.log.txt.gz | head -1 |  awk -v RS=\; '{ print NR,$0}'
1 num
2 date
3 time
4 src
5 dst
6 proto
7 service
8 action
9 xlatesrc
10 xlatedst
11 peer gateway
12 orig
13 type
14 alert
15 i/f_name
16 i/f_dir
17 product
18 log_sys_message
19 origin_id
20 ProductFamily
21 rule
22 rule_uid
23 rule_name
24 service_id
25 NAT_rulenum
26 NAT_addtnl_rulenum
27 s_port
28 xlatedport
29 xlatesport
30 message_info
31 inzone
32 outzone
33 ICMP
34 ICMP Type
35 ICMP Code
36 TCP packet out of state
37 tcp_flags
38 scheme:
39 methods:
40 encryption failure:
41 partner
42 community
43 fw_subproduct
44 vpn_feature_name
45 srckeyid
46 dstkeyid
47 IKE:
48 CookieI
49 CookieR
50 msgid
51 IKE notification:
52 Certificate DN:
53 IKE IDs:
54 user
55 rule_guid
56 hit
57 policy
58 first_hit_time
59 last_hit_time
60 log_id
61 message
62 ip_id
63 ip_len
64 ip_offset
65 fragments_dropped
66 during_sec
67 fw_message
68 reject_category
69 DCE-RPC Interface UUID
70 Log delay
71 description
72 status
73 version
74 comment
75 update_service
76 Protection Name
77 Severity
78 Confidence Level
79 protection_id
80 SmartDefense Profile
81 Performance Impact
82 Industry Reference
83 Protection Type
84 detected port
85 protocol
86 Attack Info
87 attack
88 FollowUp
89 Log ID
90 spi
91 encryption fail reason:
92 rpc_prog

One-liner: How to convert CheckPoint firewall logs

To make the log format predictable, create /etc/fw/conf/logexport.ini with the following

For R70 (Secuplat):

[Fields_Info]
included_fields=num,date,time,src,dst,proto,service,action,xlatesrc,xlatedst,peer gateway,<REST_OF_FIELDS>

For R77 (GAIA):

[Fields_Info]
included_fields=date,time,src,dst,proto,service,action,xlatesrc,xlatedst,peer gateway,<REST_OF_FIELDS>

Create a directory for the converted logs:

mkdir /var/log/2019.txt

Run the following command to convert all logs, for example, for January 2019:

 
for i in $FWDIR/log/2019-01-*.log; do echo $i; fwm logexport -n -p -i $i |  gzip -c - > /var/log/2019.txt/$i.txt.gz; done

One-liner: how to get Pulse Secure (Juniper SSL VPN) ACLs per role

Export ACLs as XML from the GUI into acls.xml file, then run the following command:

cat acls.xml | sed -e 's/&amp;/\&/g' | awk '
BEGIN { acls=""; acl=0;}
/<network-connect-acl>/ { acl=1;}
/<resource>/ { if ( acl == 1) {a=gensub(".*<resource>(.*)<\/resource>.*","\\1","g",$0); acls = acls " " a; } } 
/<roles>/ { if (acl == 1) { roles[gensub(".*<roles>(.*)<\/roles>.*","\\1","g",$0)]=acls;} }
/<action>/ { if (acl == 1) { action=gensub(".*<action>(.*)<\/action>.*","\\1","g",$0)};}
/<\/network-connect-acl>/ { acl=0; acls=""; for ( i in roles) {printf("%s;%s;%s\n", i,roles[i],action);} delete roles;}' | sed -e 's/; /;/g' | less

The result will be shown in the following (semi-colon separated form):

Role;ACl1 ACL2 ACL3 …;action

One-liner: how to check temperature differences

The idea of this not-quite-one-liner is to periodically check the DIMM temperatures and show the differences (works in KSH, and does not work in BASH):

n=1

# Check all DIMM temperatures 
for temp in `sensors | fgrep DIMM | sed -e 's/°C.*//' -e 's/^.*+//' -e 's/\./,/'`
do

	(( B$n=$temp )) # temperature in the beginning
	(( D$n=$temp )) # current temperature
	(( n+=1 )) 	 	# next DIMM
done

while true
do

	sleep 10
	n=1
	for temp in `sensors | fgrep DIMM | sed -e 's/°C.*//' -e 's/^.*+//' -e 's/\./,/'`
	do
		(( temp_var=D$n ))
		(( DIFF1=$temp-$temp_var )) #temp diff from the last iteration
		(( temp_var=B$n ))
		(( DIFF2=$temp-$temp_var )) #temp diff from the beginning
		/bin/echo -e "$temp \t $DIFF2 \t $DIFF1"
		(( D$n=$temp ))
		(( n+=1 ))
	done
	echo ----------------------------------------
done

The result:

72,0 	 0 	 0
65,0 	 0 	 0
65,0 	 -0,5 	 -0,5
63,0 	 0 	 0
72,5 	 0 	 0
72,5 	 0 	 0
75,5 	 0 	 0
64,0 	 0 	 0
----------------------------------------
72,0 	 0 	 0
65,0 	 0 	 0
65,0 	 -0,5 	 0
63,0 	 0 	 0,5
72,5 	 0 	 0
72,5 	 0 	 0
75,5 	 0 	 0
64,0 	 0 	 0
----------------------------------------
71,0 	 -1 	 -1
65,0 	 0 	 0
65,0 	 -0,5 	 0
62,5 	 -0,5 	 -0,5
72,5 	 0 	 0
72,5 	 0 	 0
75,5 	 0 	 0
64,0 	 0 	 0
----------------------------------------

One-liner: how to do a ping sweep in bash

Ping sweep from 10.0.1.1 to 10.0.1.31:

 for i in {1..31}; do ping -c 1 10.0.1.$i | fgrep ttl & done 2>/dev/null | sed -e 's/^.*from //' -e 's/:.*$//' | sort -n -t. -k4

Result:

10.0.1.1
10.0.1.2
10.0.1.3
10.0.1.7
10.0.1.9
10.0.1.10
10.0.1.12
10.0.1.13
10.0.1.14
10.0.1.15
10.0.1.16
10.0.1.17
10.0.1.18
10.0.1.19
10.0.1.20

One-liner: how to search media files for file and track names

Let’s find all media files with file or track names containing the word “brain” recursively beginning from the current directory:

find . -type f -exec mediainfo '{}' \+ | tr -s ' ' | tr -s '      ' | sed -e 's/ : /:/' | awk -F: 'function aprt () {printf("%s | %s | %s\n",a["Complete name"], a["Duration"], a["Track name"]);a["Complete name"]="";a["Duration"]="";a["Track name"]="";} {a[$1]=$2} /^General/ {aprt()} END {aprt()}' | fgrep -i brain

Result (file name | duration | track name):

./Jazz Side of the Moon/8 Brain Damage.flac | 6 min 20 s | 
./Pink Floyd - TDSOTM - (CP35-3017 Japanese Black Triangle)/8 Brain Damage.flac | 3 min 50 s | Brain Damage
./The Dark Side of The Moon/09.flac | 3 min 51 s | Brain Damage
./Vocomotion - Dark Side of the Moon/08 - Vocomotion - Brain Damage.mp3 | 3 min 53 s | Brain Damage

The “one-liner” above could have been simpler and faster, if there were no spaces in the file names.