Malware Incident Response pt. 2. (Collecting Volatile and Non-Volatile data)

Introduction 

During a live incident response, you have two subsets of data to collect from. There is volatile & nonvolatile. Volatile data is any data that requires a power source to maintain its existence. So in other words, we are talking about data that exists on a computer that is in a running state, that won't exist, in its entirety, if the computer is turned off. Volatile data exists primarily within the memory of a computer. As we all know, RAM will not maintain its contents, if the system has been powered down. Memory is truly a treasure trove of information. When you want to know what programs have been run in the past or is currently running, any connections that have been made either locally or remotely, IP addresses, passwords, bypassing encryption, log entries, etc. This information isn't available to you simply because the computer is in a "running state". There's more to it than that. When performing a live acquisition, you can image a hard drive while the system and all its processes are running, but it doesn't provide the same type of data. Data you get in a hard drive is an important aspect of nonvolatile forensics, but the technology of Memory is so powerful, there are artifacts left in there that are a must have. For example: In order for obfuscated malware to run, or any encrypted program for that matter to run, the malware must first decrypt itself when running inside memory. Unless some malware author statically built all the the necessary binaries inside of the program, (unlikely), it's going to need to decrypt itself in order to run and reach out to the functions in the Windows API, via those friendly dll's. Even when a clever rootkit is able to hide it's presence from a users shell, a memory dump will reveal all the steps the rootkit was taking. To point out, malware in general  attempt anti-forensics by running all of its code within RAM, and not touching the hard drive. Malware authors know, they need to obfuscate any artifacts it may leave on a drive, or just all together not touch the drive completely.

Volatile data: Network connections are a good example of volatile data. Once you shut down the computer, all the information that is in memory that discloses what ports are opened will be lost.

Non-Volatile data: You acquire non-volatile data during a live response, just like you would when acquiring an image of volatile data from memory. You can image the hard drive or memory in a virtual environment, or while booted into the infected users account. An image of the hard drive is an example of non-volatile data, but, if you wanted to acquire non-volatile data that is forensically important, without imaging the entire drive, you would choose to selectively capture things in a running state, such as host files, prefetch files, autoruns, event logs, the registry, etc.. There are tools that are designed to specifically extract this type of data. This information as you can see, is very limited, and is only performed when imaging the drive isn't really necessary. Similarly with memory acquisitions, if a full memory capture isn't necessary, then you can before a runtime interrogation, that scans through memory for any "malicious" indicators.

All volatile artifacts found on a system, are not all created equally. Some artifacts have a shorter life expectancy than others. There is an "order of volatility", which is a helpful guideline to follow when prioritizing the different types of data an investigator should analyze first. So, if a responder were to arrive to an incident, here is the order of prioritizing ones steps, when performing a live collection of volatile data WITH an incident responders toolkit, a.k.a, a live cd, on a Windows 7 machine:

Volatile Data collection

1. Boot up into your live cd and run a trusted shell. (Ex: Helix, SIFT, or anything running Sysinternals.)
2. Check the systems date and time, by running the command "echo %DATE% %TIME%", inside the trusted command shell, and compare with an external time. If the time is off, that means someone tampered with the CMOS. So you want to check and make sure this is accurate, so later on when you perform a timeline analysis, you can attest to it's accuracy. But, if and when you're concerned about time stomping, you will further your investigation by checking event logs and checking the "sequence values" of .lnk files, during a dead analysis. 
3. Acquire the contents of memory. (This is a tricky thing to consider. Since in this scenario we are running a live cd, as we know, live cd's run entirely inside RAM, to maintain the pristine state of the drive, while at the same time, overwriting data in RAM. If you choose to use a live cd, you will have a large toolset at your disposal for collecting other volatile data on the running system, at the expense of the memory. You maybe could find a toolkit with a smaller footprint, or you could run a tool from a usb drive. While booted into the infected account, you would plug the usb drive into the system, run something like a bat file, and do an acquisition of RAM, and nothing else. For example, something like Memoryze can run its executables without installing itself on the infected users drive.)
4. Gather hostname, user, and OS details with commands: There's the "hostname" command and the "whoami" command. For system info (Win7), there is systeminfo | findstr /B /C:"OS Name" /C:"OS Version"command. 
5. Identify users logged onto the system. CLI tool from Sysinternals, Psloggedon. The native command, "net users", will also show you a list of logged on users.
6. Inspect network connections, open ports such as TCP and UDP ports, recent DNS queries, inspect NetBios name table, ARP cache and the internal routing table, and if possible, monitor remotely with wireshark or NetworkMiner: Commands include, a native "Netstat -ano", or Sysinternals 'TCPView'. Sysinternals is truly a must have in any Windows toolkit. To display DNS resolution =  "ipconfig /displaydns". Netbios = 'nbtstat -c', then the 'net' command to see what files recently traveled a netbios connection. MAC to IP cache that will show you previously connected systems = 'arp -a'. 
7. Examine running processes: A native tool = 'tasklist', or a native 'get-process' in powershell. The best third party tool is 'Process Explorer' via Sysinternals.
8. Correlate open ports to associated processes and programs: This phase of analysis assumes you have found a few programs that seem suspicious, then you sure up 'netstat -ano' once again, which will tell you what ports the malicious program is using. A third party tool called 'openports' will give same information as netstat, while also showing what path the program belongs to. The third party tool that provides the most info about a program and its port is the 'CurrPorts' from Nirsoft. 
9. Examine services and drivers: To view services, run native tool 'tasklist /svc', or a third part from Sysinternals 'psservice'. When wanting to list drivers, use the native command 'driverquery', or a third party from nirsoft = 'DriverView'.
10. Determining open files: When a malicious programing is successfully executed on a suspects system, seeing what files the program has opened may lead clues as to the programs intentions. You will have to use a third party tool for this examination = 'OpenFilesView' & 'Open Save Files View', which are both great programs from Nirsoft. It accesses two registry keys (OpenSavePidlMRU and OpenSaveMRU) which allow it to display a comprehensive list of all the files that you have opened with Windows.
11. Collect the command history: With a unix based OS, a system that uses bash in its shell will maintain a log on the system, that recorded all the commands that were typed into the system, even after the shell has been closed. Windows however, does not have logging like this enabled.Once a cmd prompt is closed, all the commands that were typed in it, is lost. If you do happen upon a computer with an open cmd, type 'donkey /history', and it will list to you all the commands typed in that session. 
12. Identifying Mapped drives & Shares: malicious code does not always have functionality to propagate through network shares, some specimens, such as the polymorphic file infector named W32/Bacalid,51 identify and affect shares on an infected system. (Malware Forensics, 2008). The native 'net' utility comes in handy once again, for identifying network shares on a system = 'net share' The native net utility is used to identify and configure any logged on users, file sharing, or to view any active network connection between users, etc.. Otherwise, use the installed psloggedon from sys internals. Again, sys internals is really your best option for an incident response, and all of the categories of volatile data are covered by their toolkit, even though i'm not losing them each time.
13. Determining Scheduled Tasks: First run the native 'at' command to see if there are any currently scheduled tasks, then run 'schtasks', and you will see a more comprehensive list of default tasks, such as normal system backup, system restore points, but nothing out of the ordinary. schtasks and net commands work both ways, they not only show you system information, but you can also use them to set up new users to a system with the 'net user' command, or schedule tasks with 'schtasks /create'.
14. Collecting clipboard contents: 'insideclipboard' by Nirsoft. Even if a remote user is logged into a system, a copy and past will render its contents into a space in memory on the system the user is logged into. 

This list of tools is by no means an extensive list. These are the bare essentials. I listed them within particular areas of volatility, simply as an introduction This list doesn't even include the scripting powers of powershell, nor the fantastic perl scripts that work great for incident response, made available by Harlan Carvy and his Perl Scripting book/books. I'm also just giving a brief overview because i'm planning on putting these tools into action in future posts.  

Non-Volatile Data Collection (if a full drive image is not feasible):

1. Identify System Security Configurations: collect the patch level information with 'Winupdatelist', logging settings (security logging) with 'auditpol', and access control lists with 'dumps'. You can also try 'Microsoft Baseline Security Analyzer'.
2. Identify Trusted Host Relationships: Most computers, when wanting to connect to the internet, will use a DNS server, and save a DNS cache local to the system. The purpose of a DNS server, is to pair a website (hostname), to an IP address, for faster easier connections. Otherwise, you would have to type the IP of an address to visit it, instead of simply typing www.google.com. Other than websites, each computer system has a hostname that is paired with your IP address, and if you have file sharing turned on, it is open for remote connections. Each computer has what is call a 'hosts file'. On a Windows 7 machine, it is located at %windir%\system32\drivers\etc\. Since most computers these days rely on DNS connections, this host file is rarely change, and/or rarely relied upon by the computer. But, every now and again, malware can target this file, for redirecting traffic, or to simply block certain websites that an AV or systems update will rely upon. There will be a text file at that paths location that you will want to copy to any external storage. 
3. Inspect prefetch files: The Windows OS creates a prefetch file whenever an application is run for the first time, and is located at the '%SystemRoot%\prefetch' directory, appended with the extension “.pf.” So if an application has been run on the system, it creates a file that is paired with that app, and saves code in its directory, to help the application load faster the next time you want to run it. So if a malicious app was recently run, a .pf file would be listed in that directory. But say if an attacker used the systems already installed netcat program, which was installed 10 months ago? That information wouldn't tell me much. But the .pf also contains some beautiful meta data. So there isn't just a time stamp that tells us when it was first installed, no, there is also metadata that tell us when it was last run. The prefetch file itself contains a 64-bit time stamp indicating when it was last run, as well as a count of how many times the application has been run. (Digital Forensics with Open Source tools. 2011). If you want to know the offsets to these identifiers, it's mentioned in the book. To list the prefetch directory, run the commands you see in the figure below:

4. Inspect Auto-Start locations: When a system boots, there are programs configured to start automatically when the system runs, and these settings are maintained with the systems Registry. You can use 'autoruns', from sysinternals, or utilize WMI by running, 'wmic startup get caption, command'. But of course, Sysinternals will provide you with a lot more information. 

5. Collect Event Logs (Windows Event Logs): Application Logs, Systems Logs, and Security Event Logs, are turned on by default in Win7. There are many other types of logs that exist in Windows 7, but these are the ones worth pointing out now for the time being. There is a built in utility that allows you to view these logs called the "Event Viewer", (type 'eventvwr' into the command prompt). Depending on how active you are on your system, there usually is a ton of event records in these files. The security even log is one that will tell you the login attempts that were made against your system or network. There is another utility in Windows 7 professional (not basic editions) that allows you to configure what you want recorded to these event logs, and that utility is call the 'Local Security Policy Editor'. It is here where you can enable The Security Event Log to record all of the login attempts to your network, and or any access to any resource or file shares that have been enabled. The Application log provides records that list any errors or crashes that occur in an application (troubleshooting purposes). The System log records errors that relate to the system. If you're interested in Anti-Virus logs, you can look under the 'Applications & Services Logs', shown in the figure below. I don't have an AV installed on my VM, but i know that is where it would be located.

Event logs work best when applied to creating a TimeLine. So, in the event of responding to an incident, in order to collect these details, it is best to use the 'PsLogList', from sysinternals. Books are mentioning 'eldump', but i like sysinternals. No offense Denmark. Use 'Net view" to list all members of a given workgroup. To display information about group policies applied to the users account type 'gpresult /Scope User /v'. 'Net user' lists all the accounts assigned to the local system.

6. Finding hidden files: The main priority of a Rootkit is to hide its files and processes, while evading detection. They want to trojan in files and programs, then make sure they stay well and hidden. A good tool that can reveal that a rootkit has hidden some files is the 'Mcafee Rootkit Detective'. Certain files and directories are normally hidden for your protection, and to display them, open explorer, click the organize button in the top left, click folder and search options, click the view tab, then select 'show hidden files, folders, and drives'.

This of course will not show you any files hidden by a Rootkit, and trying to discover if a rootkit is installed on a system, and/or try to  identify rootkit activities is very difficult to do without a third party tool. The closest you can get to a manual inspection involves a static analysis of a memory dump, or a dynamic analysis of the functions and api calls, depending on your available resources. Rootkits will hide processes, services, handles, threads, registry keys, and network connections, especially installed packet sniffers that have been put in promiscuous mode. Rootkits will also alter function calls, implement API hooking and dll injections. A rootkit issue, is a kernel deep compromise. Rootkits are a remarkably complex piece of malicious code. Their abilities are very fascinating, and very technical, so i suggest picking up "The Rootkit Arsenal", by Bill Blunden.

Aside from hiding techniques enforced by rootkits, there are many other places where malware can and will hide. The three main hiding techniques that come to mind are Alternate Data streams, Steganography, and/or hiding in a File Systems Slack Space. ADS's is a feature for NTFS filesystems, and allows files and registry keys to be hidden. More specifically, ADS's allow you to hide either data, scripts, malware, or executable code inside the $DATA attribute of a file. Not the default $DATA attribute that all files have, but when a file creates a second $DATA attribute for itself, that second $DATA attribute is essentially the alternate data stream. The reason why windows created this technology was so that Windows could be compatible with HFS+. Not only for compatibility purposes, Windows applications will use an ADS in order to to provide a summary of details about the file, or provide an image thumbnail preview. To create an ADS, in the 'Run" utility, type cmd to start it up. Change your director to your documents, then type 'echo "Hello There" > file.txt:foo'. The ':' operator tells the OS to append an ADS to a new file called 'file' will be listed in your documents, but when you open it, thee will be nothing inside of it. 'foo' is just the name we are deciding to give the stream. It could be anything you want to name it, ex: 'secret' instead of 'foo'. But whatever you list there after the colon, will be what is viewable if and when you run an executable from an ADS.
See below:








































Notepad is empty. But if you type 'dir /r', you will see that there are two files existing in the directory.



















If you want to put an executable inside of an ADS, you will have to use the 'type' command. So lets put solitaire.lnk inside of test.txt:secret.exe (this is our alternate data stream that we will be creating). Since solitaire.lnk is already installed on Win7 by default, its path location is 'C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Games', so we have to 'cd' to this directory in order to put solitaire.lnk into our ADS.

Oops, in order to run the 'type' command, you need to run as admin. So right click cmd, run as admin, then type the commands that are listed below in order to get into your user directory:













Now we need to 'cd' to the directory where solitaire.lnk is located by typing 'cd C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Games'. Once inside that directory, now we can add our executable into our ADS that we will name test.txt:secret.exe, by typing the following command: 'type solitaire.lnk > text:secret.exe', then press enter. Now solitaire.lnk will have been placed inside the test.txt:secret.exe file.
Lets run both 'dir' and 'dir /r', and you will see our newly created ADS:

That solitaire.lnk could have just as easily be a malicious executable, where an attacker placed his malware. Just imagine the 'secret.exe is a malicious file. So if we were move the test.txt with the solitaire.lnk executable inside of it, say, to our desktop, we should be able to run solitaire from there. To move this file, make sure you're in the same directory as before, and type 'move test.txt c:\Users\Aaron\Desktop', press enter, and you will see our file moved the the desktop. Then change your directory to your desktop to run the executable file. 

(*** Update: well up until this point, i am unable to get the solitaire.lnk executable to run. I've tried the command 'start test.txt:secret.exe:$DATA', and it says unable to find an executable. Come to find out, you cannot run an .exe from inside an ADS unless you create a symbolic link between solitaire.lnk and test.txt:secret.exe by typing the command 'mklink solitaire.lnk test.txt:secret.exe'. I don't know if this will work. Maybe in the future i will test it out. But for now, i really have to continue moving forward.

I've kind of veered off track here a little. So recap. To find alternate data streams with a 3rd party tool, i suggest 'Streams', from sysinterals, or try 'lads.exe', from heysoft. Since i brought up rootkits, if you want to try and see if one is installed, you will need to run from a trusted shell, since rootktis will alter any tools that you try and run from the suspects operating system. Not all third party tools will uncover a hidden rootkit. For example, the famous Hackdefender rootkit couldn't be displayed in a Helix live cd, but could be uncovered with Prodiscover, but then again, there were certain hidden processes and network connections that Prodiscover you couldn't discover. To find a rootkit, it is more efficient to use more than one tool designed to locate hidden rootkits, on an infected system. 

Since rootkits can prevent certain function calls from the windows api, or cause a redirection, it would be a good idea to analyze a system at the file system level, and by that i mean all the way down to its actual data structures. Say for example you are running a forensic tool during a live analysis, and you want to do a top down scan of the entire system, a rootkit can prevent the 'findnextfile' function call for a specific file, leaving it completely hidden. So if you manually analyze the file system itself, you are not relying on any API calls :). Functions operate on these 'streams", and rootkits like to hook them to hide. So, if you were to analyze the Master File Stable, the great overseer and record keeper of every file in the file system, you will be able to parse the MFT and find any record of an ADS. (I haven't really been clear in my delivery so far. ADS's are a hiding technique that rootkits will employ, in order to stay hidden.) And parsing the matter file table is a good way to look for alternate data streams that may be injected with extra code to hide itself from a dir /s, or a forensic tool. 

The Sleuth Kit is a great tool for parsing the master file table. You can use TSK for this type analysis on either a live running system, or on an imaged copy in a postmortem investigation. I'll list the master file table and its contents on a live system, to save the time it would need to take to image the drive. You can run WinFE, (which contains the TSK toolkit), from within a USB drive, but it will load dependencies, (DLLs). I would give a step by step, but i will give example runs for all of these tools in the future. You will first have to install Cygwin, and run the TSK commands from a C:\Cygwin root.

As far as slack space is concerned, it's a little more tricky, simply because there is a large amount of it in a file system. You get slack space on a drive, when a file doesn't fill all the clusters it was assigned to. You get RAM slack, which is random data from a memory buffer, that is added to a file to pad empty spaces, which could contain artifacts from previous moments in memory., but i think in later versions of windows, it's now just a bunch of padded 0's). This padding only happens to the last sector of a drive. Drive slack is when a files slack space is filled with the unallocated drive data that used to exist there before it was deleted. Since RAM slack is so small, it is usually preferable to hide malicious files inside of the Drive Slack. It's relatively simple to carve unallocated space on a drive, but sadly slack space isn't included in unallocated space, because it is actually space that is allocated to a file. I'm not sure what methods there are to analyze ALL of the slack space that exists inside of a file system, the only way i can even this of how to analyze slack space of a single file is with TSK's 'icat', by dividing the size of the file by its implemented data unit size. 'icat with the /s and /r switch' will give you the contents of a files slack space, with any possibly deleted data'. Most people say they love Volatility, mainly because you're able to construct your own tools, but in my opinion, i love TSK. 

Another very important piece of non-volatile data that is vital to an investigation, are the MAC timestamps that each file maintains as its metadata. If the infection period if known, you could use a tool like 'macmatch.exe', to extract all the metadata from all of the files that were created or modified during that time period, and output to a text file. Or use log2timeline, create a body file, and analyze with TSK or view in a spreadsheet. 

7. Dumping and Parsing the Registry Contents: It's interesting how the "Malware Forensics, Field Guide For Windows Systems" book has listed registry dumps under the non-volatile data collection process. Just to be clear, there are certain registry paths that are volatile, while other parts of the registry are non-volatile. For example, the HKEY_CURRENT_USER & HKLM\HARDWARE are volatile hives, along with many other keys and hives that only exist in memory. As far as non-volatile registry artifacts, there are Volatility plugins that Extract SYSTEM and SOFTWARE hives from disk, and not from memory. I don't know off the top of my head which keys and hives are volatile or non-volatile. I know registry Hive files are maintained on disk, but this is not a complete form of the registry. A complete form only exists once the registry is loaded into memory. So there is a lot of information about the registry that doesn't exist as "non-volatile". I am not nearly educated enough to have a solid recollection of the windows registry. The registry is a very very vast catacomb of information. I am not even going to attempt a brief introduction into the registry in the context of volatile/non-volatile data collections. Yes you can use 'regripper.exe' or sysinternals 'dumpreg', but like i said, the registry is a reserved topic all by itself, that i will attempt sometime in the future. 

8. Examine Web Browsing Activities: Finally, there's the non-volatile data i'm sure even non-forensically inclined individuals all know about, and that is a browsers history, cache, and cookies. All are located either on disk in a Temp folder, sqlite file, in a pretty little non-volatile index.dat file, or located in some registry key location (like types URLs). There could be a fun quick discussion to be had about browser forensics, but sadly, it's late, and i need to wrap this post up. Tools you can use during a live incident response, you can use 'web historian', or you can use the many browser utilities that are available from nirsoft, ex: IEcacheViewer, IEhistoryview. 

P.S. My apologies for the bad organization of my blog posts. There are a lot of topics in forensics, and once i start writing these, i tend to try and cram as much as i can. As i said before, this post is just an introduction into the different types of forensic tools and the areas in which they are applied. These are simply the most basic types of tools an incident responder can use. There are toolkits out there that do the job a lot more efficiently. I will apply many of the tools i've just mentioned in future blog posts that i will be making shortly. As soon as i can find an executable malicious binary to run in a VM, i'll be able to test these out. I know contagio has a bunch of samples, and so does malwaremustdie. But a lot of those samples are obscure and more technical. I am hoping to find a very basic general sample of malware. I tried creating a post last week discussing as a tutorial, how to create a honeypot with honeyd and nepenthes. I would strongly suggest finding another type of honeypot to attempt a tutorial with. Those tools are very outdated, and i simply could not get them to run. I had to scratch a weeks worth of notes. I was interested to create a honeypot on an actual physical system with honeyd, but simply could not get it running. I was surprised because I was using the "Malware Analyst's cookbook", and "Applied network security monitoring", and my guides. The first book is about 5 years old, but the ladder is only 3 years old, so i'm not exactly sure why they are presenting exercises with tools that are too old to work. Oh well. Will have to find malware samples the other way. Cheers.