Troubleshooting CLI commands
Overview
This Recommended Reads provides some of the basic troubleshooting tools that can be used from Advanced Shell of Sophos.
Accessing CLI
To access the Shell, SSH into Sophos and select option 5. Device Management and 3. Advanced Shell. For detailed instructions, refer to this KBA "Sophos Firewall: How to SSH to the firewall."
Commands
-
bwmon
This command displays the bandwidth of every interface in Sophos. By default, it’s shown in Kbytes/s. You may press the following buttons on the keyboard to show different values and change the output.
'+' increases sampling time by 100ms
'-' decreases sampling time by 100ms
'u’ cycle: bytes, bits, packets, errors
't' cycle: current rate, max, sum since start, average for last 30s
-
showfw
This command will show the current Firmware version available in SFLoader.
-
nslookup
This command can be used to resolve a DNS name. Using the "-q" option, you can specify which types of DNS records you want returned in the results.
For example, nslookup –q=mx sophos.com - would return the MX (Mail Exchanger) records for the Sophos.com domain.
The DNS record types that can be requested are:
A – Host record for an IPv4 address
AAAA – Host record for an IPv6 address
CNAME – An alias for another DNS name
MX – Mail Exchanger
TXT – Used to hold data for things like SPF (Sender Policy Framework), DKIM (DomainKeys Identified Mail), and more
PTR – A pointer to a canonical name
-
top/psmon
This command provides a dynamic real-time view of a running system. It can show system summary information and a list of processes or threads currently being managed by the kernel. Top/psmon provides an ongoing look at processor activity in real time. It displays a listing of the most CPU-intensive tasks on the system and can provide an interactive interface for manipulating processes. It can sort the tasks by CPU, memory, and runtime usage.
PID - Process Id. The task's unique process ID periodically wraps, though never restarting at zero. In kernel terms, it’s a dispatchable entity defined by a 'task_struct'.
PR - Priority. The scheduling priority of the task. If you see ‘RT' in this field, it means the task is running under 'real time' scheduling priority.
NI - Nice Value. The nice value of the task. A negative nice value means priority, whereas a positive nice value means lower priority. Zero in this field means priority won’t be adjusted in determining a task's dispatchability.
VIRT - Virtual Memory Size (KiB). The total amount of virtual memory used by the task. It includes all code, data, and shared libraries, plus pages that have been swapped out and pages that have been mapped but not used.
RES - Resident Memory Size (KiB). The non-swapped physical memory a task has used.
SHR - Shared Memory Size (KiB). The amount of shared memory available to a task, not all of which is typically resident. It simply reflects memory that could be potentially shared with other processes.
S - Process Status. The status of the task can be one of the following:
D = uninterruptible sleep
R = running
S = sleeping
T = traced or stopped
Z = zombie
%CPU - CPU Usage. The task's share of the elapsed CPU time since the last screen update is expressed as a percentage of total CPU time.
%MEM - Memory Usage (RES). A task's currently used share of available physical memory.
TIME+ - CPU Time (hundredths). Total CPU time the task has used since it started. The same as 'TIME', but reflecting more granularity through hundredths of a second.
COMMAND - Command name or Command line. Show the command line used to start a task or the name of the associated program. You can toggle between the command line and name with "c", which is both a command-line option and an interactive command.
-
free
This command shows available and used RAM on the device.
- Mem shows how much RAM is currently free or used.
- Buffers indicate how much data is cached into the unused RAM.
- Swap shows you the total available swap and how much is free or used.
- All units on above output are in bytes.
-
tcpdump
This command is a packet capture tool that allows interception and capture of packets passing through a network interface, making it useful for understanding and troubleshooting network layer problems. It helps monitor packet flow coming on the interface, response for each packet, packet drop, and ARP information.
tcpdump [-n] [-i interface] [-w file] [-r file] [-s0] [-X] [-v] [expression]
-n Show IP addresses rather than hostnames
-i Listen on interface
-w Write to file in pcap format
-r Read from pcap file
-s0 Show the whole packet that doesn’t truncate
-X Show both hex and ASCII (helpful for looking at clear text errors)
-v Show verbose output (use -vv and -vvv for more verbose output)
[expression] You can use expressions to filter your query. See below for the most commonly used.
- host [ip | hostname] - filter for a specific host
- port [port number] - filter for a specific port
- ICMP - only show ICMP packets (ping and traceroute for example)
- and [not] [or] - you can use multiple expressions together with
- these keywords.
not (‘!=’ or ‘not’) Negation
and (‘&&’ or ‘and’) Concatenation
or (‘||’ or ‘or’) Alternation
Examples
To print all packets from host 192.168.1.69 arriving at or departing on eth1 (shows only IP addresses and verbose output)
tcpdump –eni eth1 host 192.168.1.69
To print all ICMP packets arriving at or departing on any interfaces
tcpdump –eni any icmp
To print all packets from host 192.168.0.121 or 192.168.0.135, excluding traffic on port 22 and port 4444 arriving or departing on eth5 (important to exclude packets of the WebAdmin-session and SSH itself)
tcpdump –eni eth5 host 192.168.0.121 or 192.168.0.135 and not port 22 and not port 4444
To print all packets to and from port 80 arriving at or departing on eth0
tcpdump –eni eth0 port 80
-
drppkt
This command displays the packets dropped by firewall rules. It’ll provide connection details and details of the packets the device processes. This will help administrators to troubleshoot errant firewall rules. You can also filter the dropped packets.
drppkt host 10.10.10.1 and port 21 (This will show all dropped packets for host 10.10.10.1 and port 21)
| How to view traffic of a ... | drppkt command | Example |
| specific host | drppkt host <ipaddress> |
drppkt host 10.10.10.1 |
| specific source host | drppkt src host <ipaddress> |
drppkt src host 10.10.10.1 |
| specific destination host | drppkt dst host <ipaddress> |
drppkt dst host 10.10.10.1 |
| specific network | drppkt net <network address> |
drppkt net 10.10.10 |
| specific source network | drppkt src net <network address> |
drppkt src net 10.10.10 |
| specific destination network | drppkt dst net <network address> |
drppkt dst net 10.10.10 |
| specific port | drppkt port <port-number> |
drppkt port 21 |
| specific source port | drppkt src port <port-number> |
drppkt src port 21 |
| specific destination port | drppkt dst port <port-number> |
drppkt dst port 21 |
| specific host for the particular port | drppkt host <ipaddress> and port <port-number> |
drppkt host 10.10.10.1 and port 21 |
| the specific host for all the ports except SSH | drppkt host <ipaddress> and port not <port-number> |
drppkt host 10.10.10.1 and port not 22 |
| specific protocol | drppkt proto ICMP
drppkt proto UDP drppkt proto TCP drppkt arp |
- conntrack
This command/tool is used to list the connections in Sophos. It will also help you identify the firewall rule ID through which this packet was processed in the Sophos. In the first example of the captured conntrack, reply-sport is 3128, the HTTP proxy port for the Sophos device from where it was taken. It means that traffic was captured and processed by the web Proxy of the Sophos device. Also, look at the fwid, which indicates the firewall rule processing the connection.
proto=tcp proto-no=6 timeout=10765 state=ESTABLISHED orig-src=10.160.24.195 orig-dst=172.217.13.195 orig-sport=53498 orig-dport=443 packets=9 bytes=2243 reply-src=10.160.24.1 reply-dst=10.160.24.195 reply-sport=3128 reply-dport=53498 packets=9 bytes=6501 [ASSURED] mark=0x8001 use=2 id=1756024256 masterid=0 fwid=2 policytype=1 user=0 luserid=0 usergp=0 webfltid=1 hotspotid=0 hotspotuserid=0 hb_src=0 hb_dst=0 dnat_done=0 proxy_flags=0 icapid=0 appfltid=1 appid=100 catid=29 appcatid=5 ips=3 ips_nfqueue=0 ips_maxsesbytes=1 inmark=0x0 brdevinindex=0 devinindex=5 devoutindex=0 devin=Port1 devout= inzone=1 outzone=2 bwid=0 upclass=0:0 dnclass=0:0 sslvpnid=0 snatid=1 cluster_node=0 gwoff=0 ctflags=0x4200840a mmflags=0x10802200 dropfix=0 src_mac=00:00:00:00:00:00 dst_mac=00:00:00:00:00:00 vlan_id=0 diffserv=0 current_state[0]=7 current_state[1]=7
In this second example, you can see that fwid is zero but the connection was still processed, which in most cases means that it was a system(Sophos) generated traffic.
proto=tcp proto-no=6 timeout=10799 state=ESTABLISHED orig-src=10.20.20.63 orig-dst=84.39.152.32 orig-sport=46340 orig-dport=80 packets=7 bytes=1352 reply-src=84.39.152.32 reply-dst=10.20.20.63 reply-sport=80 reply-dport=46340 packets=8 bytes=941 [ASSURED] mark=0x8001 use=1 id=971057744 masterid=0 fwid=0 policytype=0 user=0 luserid=0 usergp=0 webfltid=0 hotspotid=0 hotspotuserid=0 hb_src=0 hb_dst=0 dnat_done=0 proxy_flags=0 icapid=0 appfltid=0 appid=0 catid=0 appcatid=0 ips=0 ips_nfqueue=0 ips_maxsesbytes=0 inmark=0x0 brdevinindex=0 devinindex=0 devoutindex=6 devin= devout=Port2 inzone=0 outzone=0 bwid=0 upclass=0:0 dnclass=0:0 sslvpnid=0 snatid=0 cluster_node=0 gwoff=0 ctflags=0x0 mmflags=0x2000 dropfix=0 src_mac=00:00:00:00:00:00 dst_mac=00:00:00:00:00:00 vlan_id=0 diffserv=0 current_state[0]=8 current_state[1]=8
Conntrack command can be used for multiple purposes, but for this specific guide, we’ll only use this command is syntax conntrack -L, which would show or list the connections. Some common parameters and options for conntrack are:
-s, --orig-src ip Source address from the original direction
-d, --orig-dst ip Destination address from the original direction
-r, --reply-src ip Source addres from reply direction
-q, --reply-dst ip Destination address from reply direction
-p, --protonum proto Layer 4 Protocol, e.g. 'tcp'
-P, --not-protonum notproto Except Protonum(Do not use this option with 'p')
-f, --family proto Layer 3 Protocol, eg. 'ipv6'
-t, --timeout timeout Set timeout
-u, --status status Set status, e.g. ASSURED
-b, --buffer-size Netlink socket buffer size
i.e. conntrack -L -s 192.168.24.195 -d 8.8.8.8 will show connections for which Src IP is 10.160.24.195 and destination is 8.8.8.8.
proto=icmp proto-no=1 timeout=29 orig-src=10.160.24.195 orig-dst=8.8.8.8 type=8 code=0 id=1 packets=8 bytes=480 reply-src=8.8.8.8 reply-dst=10.20.20.63 type=0 code=0 id=1 packets=8 bytes=480 mark=0x8001 use=2 id=1529445248 masterid=0 fwid=2 policytype=1 user=0 luserid=0 usergp=0 webfltid=1 hotspotid=0 hotspotuserid=0 hb_src=0 hb_dst=0 dnat_done=0 proxy_flags=0 icapid=0 appfltid=1 appid=0 catid=0 appcatid=0 ips=3 ips_nfqueue=100 ips_maxsesbytes=0 inmark=0x0 brdevinindex=0 devinindex=5 devoutindex=6 devin=Port1 devout=Port2 inzone=1 outzone=2 bwid=0 upclass=0:0 dnclass=0:0 sslvpnid=0 snatid=1 cluster_node=0 gwoff=0 ctflags=0x200000a mmflags=0x10902200 dropfix=0 src_mac=00:00:00:00:00:00 dst_mac=00:00:00:00:00:00 vlan_id=0 diffserv=0 current_state[0]=8 current_state[1]=8
The description of different Conntrack fields is mentioned in this table:
| fwid | Firewall rule id |
| idp | IPS policy id |
| webfiltid | Web filter policy id |
| appfiltid | Application filter policy id |
| snatid | Source natting policy id |
| svp | sslvpn policy id |
| bwid | Bandwidth policy id |
| luserid | Live user id |
| user | User id |
| usergp | Usergroup id |
| hotspotid | ID of Hotspot from which traffic is passing |
| hotspotuserid | Hotspot User id |
| appid | Application id |
| appcatid | Application category id |
| catid | Web category id |
| inzone | zone (lan/wan/dmz) id of in interface |
| outzone | zone id of out interface |
| brinindex | bridgein device index |
| devinindex | in device index |
| devoutindex | out device index |
| iif[2] | Storing interface index for routing cache |
| ctflags | Flags -1 (ctflags) |
| mmflags | Flags-2 (mmflags) |
| scanflags | Flags-3 (scanflags) |
| proxy_flags | Flags-4 (proxy flags) |
| route_dst[2] | Caching route in conntrack (both directions) |
| current_state[2] | System snapshot number (for state change) |
| classid[2] | Bandwidth classes for traffic shaping (QoS) |
| hb_src | Heartbeat health for source |
| hb_dst | Heartbeat health for destination |
| diffserv | DSCP marking (as per firewall rule) |
Usage: conntrack [commands] [options]
Commands:
-L [table] [options] List conntrack or expectation table
-G [table] parameters Get conntrack or expectation
-D [table] parameters Delete conntrack or expectation
-R [table] parameters Reclaim conntrack
-I [table] parameters Create a conntrack or expectation
-U [table] parameters Update a conntrack
-E [table] [options] Show events
-F [table] Flush table
-C [table] Show counter
-S Show statistics
Tables: conntrack, expect, dying, unconfirmed
Conntrack parameters and options:
-n, --src-nat ip source NAT ip
-g, --dst-nat ip destination NAT ip
-j, --any-nat ip source or destination NAT ip
-m, --mark mark Set mark
-k, --inmark-outmark inmark-outmark Set inmark and outmark
-Z, --inzone-outzone inzone-outzone Set inzone and outzone
-c, --secmark secmark Set selinux secmark
-e, --event-mask eventmask Event mask, eg. NEW,DESTROY
-z, --zero Zero counters while listing
-o, --output type[,...] Output format, eg. xml
-x, --g-ident Garner Ident Ex. firewall
-y, --g-serv Garner Server Ex. /tmp/garner.sock
-l, --label label[,...] conntrack labels
Expectation parameters and options:
--tuple-src ip Source address in expect tuple
--tuple-dst ip Destination address in expect tuple
Updating parameters and options:
--label-add label Add label
--label-del label Delete label
Common parameters and options:
-s, --src, --orig-src ip Source address from original direction
-d, --dst, --orig-dst ip Destination address from original direction
-r, --reply-src ip Source addres from reply direction
-q, --reply-dst ip Destination address from reply direction
-p, --protonum proto Layer 4 Protocol, eg. 'tcp'
-P, --not-protonum notproto Except Protonum(Do not use this option with 'p')
-f, --family proto Layer 3 Protocol, eg. 'ipv6'
-t, --timeout timeout Set timeout
-u, --status status Set status, eg. ASSURED
-w, --zone value Set conntrack zone
--orig-zone value Set zone for original direction
--reply-zone value Set zone for reply direction
-b, --buffer-size Netlink socket buffer size
--mask-src ip Source mask address
--mask-dst ip Destination mask address