diff --git a/modules/generators/structured_content/hackerbot_config/live_analysis_v2/templates/live_evidence_collection.md.erb b/modules/generators/structured_content/hackerbot_config/live_analysis_v2/templates/live_evidence_collection.md.erb index 2339dee7b..cdd1dc620 100644 --- a/modules/generators/structured_content/hackerbot_config/live_analysis_v2/templates/live_evidence_collection.md.erb +++ b/modules/generators/structured_content/hackerbot_config/live_analysis_v2/templates/live_evidence_collection.md.erb @@ -8,6 +8,8 @@ After suspecting a compromise, before powering down the server for offline analy ssh <%= $compromised_server_ip %> ``` +> Because the same users exist on both systems you can leave off the user name (normally *username*@*server_ip*) + **On the compromised_server VM:** To keep a record of what we are doing on the system, start the script command: ```bash @@ -20,11 +22,11 @@ script -f /tmp/evid/invst.log Consider the advantages of *handwritten* documentation of what investigators are doing. -Many of the commands used to investigate what is happening on a system are standard Unix commands. However, it is advisable to run these from a read-only source, since software on your system may have been tampered with. Also, using read-only media minimises the changes made to your local filesystem, such as executable file access times. - ### Using a Live CD/DVD -Next you will configure the compromised VM to have access to the FIRE (Forensic and Incident Response Environment) CD/DVD ISO (which is equivalent to inserting the optical disk into your server's DVD-tray). FIRE is an example of a Linux Live Disk that includes tools for forensic investigation. In addition to tools to boot to a version of Linux for offline investigation of evidence, the disk contains Linux tools for live analysis. +Many of the commands used to investigate what is happening on a system are standard Unix commands. However, it is advisable to run these from a read-only source, since software on your system may have been tampered with. Also, using read-only media minimises the changes made to your local filesystem, such as executable file access times. + +Next you will configure the compromised_server VM to have access to the FIRE (Forensic and Incident Response Environment) CD/DVD ISO (which is equivalent to inserting the optical disk into your server's DVD-tray). FIRE is an example of a Linux Live Disk that includes tools for forensic investigation. In addition to tools to boot to a version of Linux for offline investigation of evidence, the disk contains Linux tools for live analysis. Add the FIRE IR CD disk: @@ -48,118 +50,152 @@ ldd /bin/ls Examine the output, and determine how many external libraries are involved. -**On the compromised VM (Redhat7.2)**: The FIRE disk contains a number of statically compiled programs to be used for investigations. ==Check that these are indeed statically linked:== +**On the compromised_server VM ssh)**: The FIRE disk contains a number of statically compiled programs to be used for investigations. + +==Look at the commands available:== ```bash -ldd /mnt/cdrom/statbins/linux2.2_x86/ls +ls /media/cdrom0/statbins/linux2.2_x86/ +``` + +==Check that these are indeed statically linked:== + +```bash +ldd /media/cdrom0/statbins/linux2.2_x86/ls ``` ==Compare the output to the previous command== run on your own Desktop system. The output will be distinctly different, stating that the program is not dynamically compiled. -Note that, although an improvement, using statically linked programs such as these still do not guarantee that you can trust the output of the programs you run. Consider why, and make a note of this. +Note that, although an improvement, using statically linked programs such as these still do not guarantee that you can trust the output of the programs you run. ==Consider why, and make a note of this in your LogBook.== + +## First look around + +Run ls to view the contents of the home directory on the compromised_server. + +```bash +cd +ls +``` + +Run the static version +``` +/media/cdrom0/statbins/linux2.2_x86/ls +``` + +# TODO! u_r_powned-hme +Note the presence of a "u_r_powned" file in the output from the live disk version of ls! Running the local version of ls is not accurately reporting the files that exist! Lucky we thought to run another copy of ls. ## Collecting live state manually -The next step is to use tools to capture information about the live system, for later analysis. One approach to storing the resulting evidence is to send results over the network via Netcat or SSH, without storing them locally. This has the advantage of not changing local files, and is less likely to tip off an attacker, rather than storing the evidence on the compromised machine. +The next step is to use various tools to capture information about the live system, for later analysis. One approach to storing the resulting evidence is to send results over the network via Netcat or SSH, without storing them locally. This has the advantage of not changing local files, and is less likely to tip off an attacker, rather than storing the evidence you are collecting on the compromised machine itself. + +**On your Desktop VM (not from the console still sshed into the server)**, ==create a directory for the evidence you are about to collect.== + +```bash +mkdir evidence +``` + +### Saving output from the compromised server to your desktop + +**On the desktop VM (not sshed)**, ==test sending the results of some commands over SSH to your Desktop VM==: + +```bash +ssh <%= $compromised_server_ip %> "echo this command is running on the server" + +ssh <%= $compromised_server_ip %> "echo this command is running on the server" > evidence/test_output + +ls evidence + +cat evidence/test_output +``` + +> Take the time to make sure you understand which system each command above is running on. ### Comparing process lists -**On your Desktop VM**, check ensure the local SSH server (sshd) is running on your system. - -**On the compromised VM (Redhat7.2)**, test sending the results of some commands (process lists using ps) over SSH to your Desktop VM: - -> Note: if the VM is not using a UK keyboard layout, the @ and " symbols may be reversed, and the | symbol is located at the \~. Alternatively, run `loadkeys uk` in the RedHat VM to swap to a UK keyboard layout +Collect results of a process listing using ps over SSH to your Desktop VM: ```bash -ssh <%= $main_user %>@*desktop-IP-address* "mkdir evidence" - -ps aux | ssh <%= $main_user %>@*desktop-IP-address* "cat > evidence/ps_out" +ssh <%= $compromised_server_ip %> "ps aux" > evidence/local_ps_output ``` -> (Where *desktop-IP-address* is the IP address of your *desktop VM*, which should be on the same subnet as your compromised VM) - -==LogBook question: Why might it not be a good idea to ssh to your own account (if you had one on a Desktop in real life) and type your own password from the compromised system? What are some more secure alternatives?== - **On your Desktop VM**, find the newly created files and view the contents. > Hint: you may wish to use the Dolphin graphical file browser, then navigate to "/home/<%= $main_user %>/evidence". -**On the compromised VM (Redhat7.2)**, run the statically compiled version of ls from the incident response disk to list the contents of /proc (this is provided dynamically by the kernel: a directory exists for every process on the system), and once again send the results to your Desktop VM... - -First, to save yourself from having to type `/mnt/cdrom/statbins/linux2.2_x86/` over and over, save that value in a Bash variable: - -```bash -export static="/mnt/cdrom/statbins/linux2.2_x86/" -``` - -Now, to run the statically compiled version of ls, you can run: - -```bash -$static/ls -``` +Run the statically compiled version of ls from the incident response disk to list the contents of /proc (this is provided dynamically by the kernel: a directory exists for every process on the system), and once again send the results to your Desktop VM... Run the command: ```bash -$static/ls /proc | ssh <%= $main_user %>@*desktop-IP-address* "cat > evidence/proc_ls_static" +ssh <%= $compromised_server_ip %> "/media/cdrom0/statbins/linux2.2_x86/ls /proc" > evidence/proc_ls_static ``` -**On your Desktop VM**, find the newly created files and compare the list of pids (numbers representing processes) output from the previous commands. This is the second column of output in the ps\_out, with the numbers in proc\_ls\_static. +**On your Desktop VM**, find the newly created files and ==compare the list of pids (numbers representing processes) output from the previous commands==. This is the second column of output in the ps\_out, with the numbers in proc\_ls\_static. Hint: you can do the comparison manually, or using commands such as "cut" (or [*awk*](http://lmgtfy.com/?q=use+awk+to+print+column)), "sort", and "diff". For example, `cat ps_out | awk '{ print $4 }'` will pipe the contents of the file ps\_out into the awk command, which will split on spaces, and only display the fourth field. Ensure this is displaying the list of pids, if not try selecting a different field. You could pipe this through to "sort". Then save that to a file (by appending " > pids\_ps\_out"). We have covered how to use diff previously. Remember "man awk", "man sort", and "man diff" will tell you about how to use the commands (and Google may also come in handy). Are the same processes shown each time? If not, that is very suspicious, and likely indicates a break-in, and that we probably shouldn't trust the output of local commands. -### Gathering live state using statically compiled programs +c## Gathering live state using statically compiled programs -**On the compromised VM (Redhat7.2)**, save a copy of a list inodes of removed files that are still open or executing: +**On the compromised_server VM**, save a copy of a list of inodes of removed files that are still open or executing: ```bash -$static/ils -o /dev/hda1 | ssh <%= $main_user %>@*Desktop-IP-address* "cat > evidence/deleted_out" +ssh <%= $compromised_server_ip %> "/media/cdrom0/statbins/linux2.2_x86/ils -o /dev/hda1" > evidence/deleted_out ``` > Tip: on VMware VMs, you may need to replace "hda1" with "sda1". +> Also, there might not be any output if there are no deleted files being accessed. Save a list of the files currently being accessed by programs: ```bash -$static/lsof | ssh <%= $main_user %>@*Desktop-IP-address* "cat > evidence/lsof_out" +ssh <%= $compromised_server_ip %> "/media/cdrom0/statbins/linux2.2_x86/lsof" > evidence/lsof_out ``` **On your Desktop VM**, open evidence/lsof\_out. -==LogBook question: Are any of these marked as "(deleted)"? How does this compare to the ils output? What does this indicate?== +==LogBook question: Are any of these marked as "(deleted)"? If so, how does this compare to the ils output? What does this indicate?== (The answer may be that they are the same.) -**On the compromised VM (Redhat7.2)**, +**On the compromised_server VM**, Save a list of network connections: ```bash -$static/netstat -a | ssh <%= $main_user %>@*Desktop-IP-address* "cat > evidence/netstat_out" +ssh <%= $compromised_server_ip %> "netstat -a" > evidence/netstat_out + +ssh <%= $compromised_server_ip %> "/media/cdrom0/statbins/linux2.2_x86/netstat -a" > evidence/netstat_static_out ``` > (Some commands such as this one may take awhile to run, wait until the Bash prompt returns) Save a list of the network resources currently being accessed by programs: ```bash -$static/lsof -P -i -n | ssh <%= $main_user %>@*Desktop-IP-address* "cat > evidence/lsof_net_out" +ssh <%= $compromised_server_ip %> "/media/cdrom0/statbins/linux2.2_x86/lsof -P -i -n" > evidence/lsof_net_out ``` Save a copy of the routing table: ```bash -$static/route | ssh <%= $main_user %>@*Desktop-IP-address* "cat > evidence/route_out" +ssh <%= $compromised_server_ip %> "/media/cdrom0/statbins/linux2.2_x86/route" > evidence/route_out ``` Save a copy of the ARP cache: ```bash -$static/arp -a | ssh <%= $main_user %>@*Desktop-IP-address* "cat > evidence/arp_out" +ssh <%= $compromised_server_ip %> "/media/cdrom0/statbins/linux2.2_x86/arp -a" > evidence/arp_out ``` Save a list of the kernel modules currently loaded (as reported by the kernel): ```bash -$static/cat /proc/modules | ssh <%= $main_user %>@*Desktop-IP-address* "cat > evidence/lsmod_out" +ssh <%= $compromised_server_ip %> "/media/cdrom0/statbins/linux2.2_x86/cat /proc/modules" > evidence/lsmod_out +``` + +Save a copy of the Bash history: + +```bash +ssh <%= $compromised_server_ip %> "sudo /media/cdrom0/statbins/linux2.2_x86/cat /root/.bash_history" > evidence/bash_history ``` **Creating images of the system state** @@ -167,12 +203,12 @@ $static/cat /proc/modules | ssh <%= $main_user %>@*Desktop-IP-address* "cat > ev We can take a snapshot of the live state of the computer by dumping the entire contents of memory (what is in RAM/swap) into a file. On a Linux system /proc/kcore contains an ELF-formatted core dump of the kernel. Save a snapshot of the kernel state: ```bash -$static/dd if=/proc/kcore conv=noerror,sync | ssh <%= $main_user %>@*Desktop-IP-address* "cat > evidence/kcore" +ssh <%= $compromised_server_ip %> "/media/cdrom0/statbins/linux2.2_x86/dd if=/proc/kcore conv=noerror,sync" > evidence/kcore ``` Next, we can copy entire partitions to our other system, to preserve the exact state of stored data, and so that we can conduct offline analysis without modifying the filesystem. -Start by identifying the device files for the partitions on the compromised system (Redhat7.2): +Start by identifying the device files for the partitions on the compromised system: ```bash df @@ -182,12 +218,12 @@ Note that on this system the root partition (mounted on "/"), is /dev/hda1. > Help: on VMware VMs only, you may need to replace "hda1" with "sda1". -Then, copy byte-for-byte the contents of the root ("/") partition (where /dev/hda1 was identified from the previous command: +Then **you could** (see the tip below) copy byte-for-byte the contents of the entire root ("/") partition over the network (where /dev/hda1 was identified from the previous command): ```bash -$static/dd if=/dev/hda1 conv=noerror,sync | ssh <%= $main_user %>@*Desktop-IP-address* "cat > evidence/hda1.img" +ssh <%= $compromised_server_ip %> "/media/cdrom0/statbins/linux2.2_x86/dd if=/dev/hda1 conv=noerror,sync" > evidence/hda1.img ``` -> Tip: Running this will take some time, so you may wish to continue with the next step while the copying runs. +> Tip: Feel free to skip this step. Running this will take some time, so you may wish to continue with the next step while the copying runs. This command could be repeated for each partition including swap partitions. For now, let's accept that we have all we need. @@ -197,22 +233,22 @@ This command could be repeated for each partition including swap partitions. For ls -la /home/<%= $main_user %>/evidence ``` -At this stage look through some of the information you have collected. For example: +At this stage take a closer look through some of the information you have collected. For example: ```bash less /home/<%= $main_user %>/evidence/lsof_net_out ``` -Examine the contents of the various output files and identify anything that may indicate that the computer has been compromised by an attacker. Hint: does the network usage seem suspicious? +==LogBook Task:== Examine the contents of the various output files and identify anything that may indicate that the computer has been compromised by an attacker. Hint: does the network usage seem suspicious? ### Collecting live state using scripts As you may have concluded from the previous tasks, manually collecting all this information from a live system can be a fairly time consuming process. Incident response data collection scripts can automate much of this process. A common data collection script "linux-ir.sh", is included on the FIRE disk, and is also found on the popular Helix IR disk. -**On the compromised VM (Redhat7.2)**, have a look through the script: +**On the compromised_server VM (ssh console tab from earlier)**, have a look through the script: ```bash -less /mnt/cdrom/statbins/linux-ir.sh +less /media/cdrom0/statbins/linux-ir.sh ``` Note that this is a Bash script, and each line contains commands that you could type into the Bash shell. Bash provides the command prompt on most Unix systems, and a Bash script is an automated way of running commands. This script is quite simple, with a series of commands (similar to some of those you have already run) to display information about the running system. @@ -224,9 +260,7 @@ Exit viewing the script (press q). Run the data collection script, redirecting output to your Desktop VM: ```bash -cd /mnt/cdrom/statbins/ - -./linux-ir.sh | ssh <%= $main_user %>@*Desktop-IP-address* "cat > evidence/ir_out" +ssh <%= $compromised_server_ip %> "/media/cdrom0/statbins/linux-ir.sh" > evidence/ir_out ``` **On your Desktop VM**, have a look at the output from the script: @@ -245,10 +279,10 @@ Detecting the presence of rootkits is tricky, and prone to error. However, there Chkrootkit is a Bash script that performs a number of tests for the presence of various rootkits. -**On the compromised VM (Redhat7.2)**, have a quick look through the script, it is much more complex than the previous linux-ir.sh script: +**On the compromised_server VM (ssh)**, have a quick look through the script, it is much more complex than the previous linux-ir.sh script: ```bash -less /mnt/cdrom/statbins/chkrootkit-linux/chkrootkit +less /media/cdrom0/statbins/chkrootkit-linux/chkrootkit ``` > Exit less @@ -278,10 +312,10 @@ which ls This should report the path to our static binary on the FIRE disk. -It is now safe to run chkrootkit[^3]: +It is now safe to run chkrootkit[^1]: ```bash -./chkrootkit-linux/chkrootkit | ssh <%= $main_user %>@*Desktop-IP-address* "cat > evidence/chkrootkit_out" +export PATH=$static:$PATH ssh <%= $compromised_server_ip %> "/media/cdrom0/statbins/chkrootkit-linux/chkrootkit" > evidence/chkrootkit_out ``` > Help: you may get a message in the terminal before you type the password. You should still type the password for the script to run. The script should not take long to run. @@ -295,36 +329,32 @@ From the output, identify files or directories reported as "INFECTED", or suspic Also, note that the .bash_history is reportedly linked to another file. -**On the compromised VM (Redhat7.2)**, investigate the Bash history: - -```bash -$static/ls -la /root/.bash_history -``` - -What does the output mean? What does this mean for the logging of the commands run by root? - At this stage you should be convinced that this system is definitely compromised, and infected with some form of rootkit. +# TODO: REMOVE? + Save a record of your activity to your Desktop VM: ```bash cat /tmp/evid/invst.log | ssh <%= $main_user %>@*Desktop-IP-address* "cat > evidence/script_log" ``` -Power down the compromised system (Redhat7.2), so that we can continue analysis offline: +**On the compromised_server VM (ssh console tab)** + +Power down the compromised system, so that we can continue analysis offline: ```bash -$static/sync; $static/sync +/media/cdrom0/statbins/linux2.2_x86/sync; /media/cdrom0/statbins/linux2.2_x86/sync ``` > If you do not know what the sync command does, on your Desktop VM, run "info coreutils 'sync invocation'" for more information. > -> Tell the oVirt Virtualization Manager to force a Power Off. +> Tell Hacktivity to force a Power Off. Why might we want to force a power off (effectively "pulling the plug"), rather than going through the normal shutdown process (by running "halt" or "shutdown")? ## Offline analysis of live data collection -Note that even though the bash\_history was not saved (as we discovered above), we can still recover commands that were run the last time the computer was running. This is possible by searching through the saved RAM (the kcore ELF dump we saved earlier). +Note that even though the bash\_history was not saved (as we discovered above), we can still recover commands that were run while the computer was running. This is possible by searching through the saved RAM (the kcore ELF dump we saved earlier). **On your Desktop VM**, run: @@ -340,18 +370,6 @@ Open the extracted strings, and look for evidence of the commands you ran before less /home/<%= $main_user %>/evidence/kcore_strings ``` -Now press the '/' key, and type a regex to search for commands you previously ran to collect information about the system. For example, try searching for "ssh <%= $main_user %>" (press 'n' for next). +Now press the '/' key, and type a regex to search for commands you previously ran to collect information about the system. For example, try searching for "statbins/linux2.2_x86" (press 'n' for next). -## What's next ... - -In the next lab you will analyse the artifacts you have collected, to determine what has happened on the system. - -**Important: save the evidence you have collected, as this will be used as the basis for the next lab.** - -ls -la /home/<%= $main_user %>/evidence you may have to be in root or without and remember when looking at the file you've created is from the outside the VM. - -[^1]: Note that the password for root has been reset for these exercises. - -[^2]: In reality, if we *knew* the system was compromised, we would likely *leave it powered off*, and move straight to offline analysis. - -[^3]: Note that it would be better to not have to include \$PATH, and only use static versions. Unfortunately, FIRE does not include statically compiled versions of all of the commands that chkrootkit requires. +[^1]: Note that it would be better to not have to include \$PATH, and only use static versions. Unfortunately, FIRE does not include statically compiled versions of all of the commands that chkrootkit requires.