Useful shell commands
http://www.greenvalleyconsulting.org/2013/07/09...mands-and-processes/
I have been collecting useful command line procedures for the Sophos UTM platform. Some I have come up with, some I have simply scraped off the forums. I never really intended on making a post about the topic so I never kept track of the contributors (sorry), however, for my clients, I decided to publish what I had with full disclosure of the source.
Deploying A Replacement UTM (Manually Updating Firmware To Current)
When helping clients replace an older model UTM with the new SG UTM, the backup of the config can not be restored until the new SG UTM’s firmware is updated to at least the level of the old UTM. This is a series of command line entries to rapid fire update the firmware.
From the new SG UTM – first turn off automatic updates by setting the update frequency to manual.
Then login to the new SG UTM and execute the following commands:
cd /var/up2date/sys
wget http://ftp.astaro.com/pub/UTM/v9/up2date/u2d-sy...09003-310011.tgz.gpg
wget http://ftp.astaro.com/pub/UTM/v9/up2date/u2d-sy...10011-311003.tgz.gpg
wget http://ftp.astaro.com/pub/UTM/v9/up2date/u2d-sy...11003-312005.tgz.gpg
wget http://ftp.astaro.com/pub/UTM/v9/up2date/u2d-sy...12005-312006.tgz.gpg
/sbin/auisys.plx –showdesc
/sbin/auisys.plx –upto “9.312006”
Of course, extend the wget lines to the current.
Renaming A RED
As of V9.3, renaming a RED is not possible with the WebAdmin. However, renaming the a RED is easy in the command line configuration utility. Become root on the UTM via the shell and type the following:
cc
red
servers@
OBJS
When ‘server@’ is typed in, the response will be a list of REDs. Make a note of the ID that begins with ‘REF_’
<RED_xxxxxxx>
name = ‘new name’
w
quit
That’s it!
Application Control Log (V9.3):
The application control log only lists the application ID. For example, this log excerpt shows app=”127″:
# tail -f /var/log/afc.log
2015:06:02-11:07:20 utm205 ulogd[22627]: id=”2017″ ….. sub=”packetfilter” name=”AFC Alert” action=”log” fwrule=”2″ outitf=”eth1″ mark=”0x307f” app=”127″ ….
What application is this rule about?
Application Flow Control uses a lookup file to translate the app number to a text name. On version 9.312, this lookup file can be found at:
/var/sec/chroot-afc/etc/afc/plugin.conf
However, Sophos includes a handy tool to make watching your logs in realtime easier. Try:
tail -f /var/log/afc.log | afc-mark-filter.pl
And now the log has the text label added to the end of the log line:
afcname=DROPBOX
Packet Filter Commands:
To View automatic firewall rules from command line
# iptables -L AUTO_FORWARD
Packet Filter
# iptables -L -nv
Clear Buffer Memory -( Useful for any linux):
free -k; sync; echo 3 > /proc/sys/vm/drop_caches; free -k
I’ve used this one in a pinch…not often tho.
Memory Leak Observation:
slabtop -d 60 -s c
Run Astaro HTTP proxy database localy
1. ssh to ASG and login with loginuser
2. su – root
3. cc set http sc_local_db [disk][mem][none] (Choose what you prefer)
4. /var/mdw/scripts/httpproxy restart
Websurfing will be extremely slow until the database has downloaded and been put into place. The time is link speed dependent.
In case it is not obvious, all the start scripts are in /var/mdw/scripts/
View the link speed for the ASG’s interfaces?
‘ifstat’.
Bandwidth usage – IFTOP
Astaro also offers the command ‘iftop’ to see the live traffic and traffic statistics.
One can see the traffic live on an interface for Source Host, Destination Host, and Ports.
The peak and accumulative traffic is also displayed.
Run ‘iftop’
Example:
root # iftop -i eth1
Host display:————————–General:
n – toggle DNS host resolution——P – pause display
s – toggle show source host———h – toggle this help display
d – toggle show destination host—–b – toggle bar graph display
t – cycle line display mode———–B – cycle bar graph average
————————————— T – toggle cummulative line totals
—————————————- Port display: j/k – scroll display
N – toggle service resolution———f – edit filter code
S – toggle show source port———-l – set screen filter
D – toggle show destination port—–L – lin/log scales
p – toggle port display—————- ! – shell command
q – quit
Sorting:
1/2/3 – sort by 1st/2nd/3rd column
< – sort by source name
> – sort by dest name
o – freeze current order
Concurrent Connections:
sysctl -w net.ipv4.netfilter.ip_conntrack_tcp_timeout_establ ished=86400
number of established connections:
less /proc/net/ip_conntrack | grep ESTA | wc -l
1907
number of all connections:
less /proc/net/ip_conntrack | wc -l
3315
number of connections with status WAIT (close_wait):
less /proc/net/ip_conntrack | grep WAIT | wc –l
39
Saving Snapshots of TOP automatically every half hour
create a cron job with,
top -b -n 1 >>/tmp/top-report.txt
An entry for each CPU core, and possibly another if the CPU(s) has hyperthreading:
cat /proc/cpuinfo
stopped and started again the HTTP proxy:
“/var/mdw/scripts/httpproxy stop” and “/var/mdw/scripts/httpproxy start”
Restarting MiddleWare:
service mdw restart
(from root)
Warning: it doesn’t cause a complete reboot, but it does cause an HA failover, interruption of any up/downloads and VoIP calls, etc.
HD
Find what is taking the space type
df –h
df will only tell you how full the disk is.
du will tell you what files/folders are using the most space
I’d recommend:
cd /var/storage
du -sh *
find the offending directories
What kind of CPU
“cat /proc/cpuinfo”?
Determine if the disk is overloaded
vmstat -d 5
or
vmstat -d | head -2 ; vmstat -d 5 | grep hda
if hda is your hard disk; sda for scsi
That should have similar output.
The ‘5’ is 5 second updates.
You’ll have to look at the differences between the lines to figure out how many IO’s you’re getting in those 5 seconds, and whether you’re saturating the disk or not.
See detailed info about your eth:
# ethtool eth1
OR
-mii-diag eth1
webadmin passwd lost
A user may use the following commands to reset the system passwords:
cc
RAW
system_password_reset
Ctrl c
Upon saving the file and exiting, the admin may immediately navigate to WebAdmin and re-specify all passwords for the system accounts of Astaro Security Linux.
DNS Flush cache option missing in V7
the current workaround is to restart the DNS proxy from the command line as root with the following command:
/var/mdw/scripts/named restart
To change version number
login as loginuser
su –
edit /etc/version
save the file
restart the ASG so the new version is displayed in Webadmin dashboard
Change NIC order
login as loginuser
su –
edit /etc/udev/rules.d/70-persistent-net.rules
save the file
restart the ASG so the new order is loaded.
Locked out – How to regain all logins
1) Shutdown the firewall and connect a screen and a keyboard to the firewall
2) Power on the firewall, wait until the GRUB-loader starts and press ‘ESC’
3) Select ‘Astaro Security Gateway 7.2’ (not previous or rescue!)
4) Press ‘e’ to edit and select the 2nd entry
5) Press ‘e’ once again and enter ‘init=/bin/bash’
6) Press ‘ENTER’ and ‘b’ to boot up
7) Now you are able to change the passwords for ‘loginuser’ and ‘root’
8) After that press CTRL + ALT + DEL to reboot the system and wait until you get the login prompt
Reset to factory settings
Login the command-line as ‘loginuser’, afterwards as ‘root’ and enter following commands to restore to factory settings:
1. cc [Press ENTER]
2. RAW [Press ENTER]
3. system_factory_reset [Press ENTER]
The system will automatically shutdown when it’s finished.
To change version number
login as loginuser
su
edit /etc/version
save the file
restart the ASG so the new version is displayed in Webadmin dashboard
Change NIC order
login as loginuser
su
edit /etc/udev/rules.d/70-persistent-net.rules
save the file
restart the ASG so the new order is loaded.
Manually Download and apply updates:
to download the updates from Astaro website
wget “the path of the file on the web”
note that the updates must be in /var/up2date/sys
to install the updates
auisys.plx –upto “the version”
Processes Causing High Load
ps ax -o pcpu,cmd|sort -n -r|head
ps ax -o rss,cmd |sort -n -r|head
List of IP That Count Toward License Count (Home/Software License):
For home license users and users of the software appliance where the number of users are counted against an allotment, the UTM occasionally rips through the accounting database and complies a count of IP's on the inside of the UTM. This process seems to be managed by:
/usr/local/bin/count_active_ip.plx --debug --showcount
This command displays the "licensed ip" addresses but does not change anything.
Leave off the --showcount and the actual licensed ip's will be updated on the UTM.
See – UTM Home Licensing Count Explained for more information
More CC Tricks
@goldy, there are some simpler ways to reset passwords than what is listed. Rather than:
cc
RAW
system_password_reset
Ctrl c
you can simply run:
cc system_password_reset
or even better:
cc passwd [mynewpassword]
this one changes the admin password in a single line, and does not reset shell password info. no need to set all password on the next system login.
@ximera, you can view tunnel status with:
cc get_ipsec_status
This outputs the status info in perl style data structures, and if you have many tunnels, becomes harder to read. Here’s a sample output:
‘REF_IpsSitFaketunnel’ => {
0 => {
0 => {
‘established’ => 0,
‘ike_auth’ => ‘psk’,
‘ike_lifetime’ => ‘7800s’,
‘ike_natt’ => 0,
‘ipsec_lifetime’ => ‘3600s’,
‘ipsec_sa’ => 0,
‘isakmp_sa’ => 0,
‘left’ => ‘192.168.0.2’,
‘leftid’ => ‘192.168.0.2’,
‘leftsubnet’ => ‘192.168.25.0/24’,
‘right’ => ‘10.0.0.1’,
‘rightid’ => ‘10.0.0.1’,
‘rightsubnet’ => ‘7.7.7.0/24’
}
},
‘all_established’ => 0,
‘established’ => 0,
‘expected’ => 1
}
}
The bold items show you the status of each SA, and of the entire tunnel.
My absolute favorite command line tool is grep. Specifically, grep -oP can be used to pull specific data out of log lines, for summary. This requires being familiar with perl regex, but is extremely powerful. For example, how do you find out what errors have been seen most commonly in the web security log today?
try this:
grep -oP ‘error=”.*?”‘ /var/log/http.log
That pulls a list of every error=”{anything}” instance out of the http log. Pipe this to some sorting commands, and we can get a nice sorted list of the most popular errors being returned by web servers:
grep -oP ‘error=”.*?”‘ /var/log/http.log|sort -n|uniq -c|sort -n
That will return something like this:
1 error=”Connection refused”
4 error=”DNS resolution timeout”
5 error=”Host not found”
23 error=”Connection timed out”
48 error=”Timeout while reading response from Server”
43703 error=””
This specific example can be helpful in identifying specific problems, like DNS problems. it also puts problems in perspective. 4 DNS timeouts out of 43k requests indicates that this probably isn’t a local problem.
The same commandscan be used in other ways, limited by your imagination (and regex knowledge). Be careful though, on extremely high volume systems, that may have problems from disk IO bottlenecks. searching like this will not help that situation improve
TCP Parameter Changes
Persistent change of ip conntrack timeout
You can change them via command line.
To see the current values:
# cc get packetfilter timeouts
packetfilter->timeouts = {
“ip_conntrack_generic_timeout” => 600,
“ip_conntrack_icmp_timeout” => 30,
“ip_conntrack_tcp_timeout_close” => 10,
“ip_conntrack_tcp_timeout_close_wait” => 60,
“ip_conntrack_tcp_timeout_established” => 86400,
“ip_conntrack_tcp_timeout_fin_wait” => 120,
“ip_conntrack_tcp_timeout_last_ack” => 30,
“ip_conntrack_tcp_timeout_max_retrans” => 300,
“ip_conntrack_tcp_timeout_syn_recv” => 60,
“ip_conntrack_tcp_timeout_syn_sent” => 120,
“ip_conntrack_tcp_timeout_time_wait” => 120,
“ip_conntrack_udp_timeout” => 30,
“ip_conntrack_udp_timeout_stream” => 180
}
To increase a value, use:
# cc set packetfilter timeouts ip_conntrack_udp_timeout 45
Btw: There is udp_timeout and udp_timeout_stream.
So if packets are send in both directions, then udp_timeout_stream is applied.
Realtime Packet Logfile:
tail -f /var/log/packetfilter.log | awk ‘{ for (i = 1; i <= 30; i++) if ( $i ~ /2014/ || $i ~ /dstip/ || $i ~ /srcip/ || $i ~ /srcport/ || $i ~ /dstport/) printf “%s “, $i; printf “\n”; }’
UTM Configuration DUMP:
Should the configuration need to be dumped, for any reason:
# cc snapshot_get
Or to include some tech support information:
# cc support_snapshot_get
To restore a backup from the command line, proceed as follows:
#Login using SSH:
#:login: loginuser
#:password: loginuser-password
#Become root: #:su -
#:password: root-password
#Check the existing backup-files: #:backup.plx -l
#Restore the backup file: #:backup.plx -n cfg_30916_1273834058 -i
With ASG versions prior to 8.200, you had to type out the full path like this: :backup.plx -i /var/confd/var/storage/snapshots/cfg_30916_1273834058
This still works with ASG 8.200 and higher, but is no longer necessary; the new “-n” option (mnemonic: read from Named backup) is much easier to use.
ABF or EBF file
In case you uploaded a .abf or .ebf backup-file use the following commands to restore the backup:
backup.plx -i <unencrypted_backup.abf> backup.plx -d <encrypted_backup.ebf>
Background information
With ASG version 8.200, the command line tool backup.plx has become very powerful. It now supports two main kinds of options:
#input control options #:(like reading from a named backup, a file, or the online storage)
#output control options #:(like add, import, dump, write to file, send per mail)
Typically, you use one input control option and one output control option per invocation (like “read in a named backup and import it to replace the online storage”). Many input and output options imply sane defaults on the other side. For example, just using the input option “-d” (decrypt) but not giving any output option, the default is to import the backup after decrypting it. There are a few special options (like list, remove, encrypt, edit). To see the full documentation of the version of backup.plx provided with your ASG, go to the shell command line and type # backup.plx -h