IR and forensic talk

I know I haven’t been really active on the blog lately (really not written a thing) but I wanted to talk about the new release of log2timeline. So version 0.61 was released few days ago. It mostly contains some bug fixes (at least on my behalf). The only real changes that I did was to add an input module to read the log2timeline CSV format. I also added  a bash_completion script that makes it easier to use the tool in the bash shell.

However this release does hopefully mark a shift in the development of the tool. This release has seven new input modules, six of which that were donated to me. And I’ve got one more module that will be part of the next release. This is at least to me very exciting news, since that means that other people are starting to use the tool and find it to be useful enough to add modules to it, and I hope that this shift in development will continue

I would like to thank an anonymous donator that contributed five input modules and Willi Ballenthin that contributed another module to the tool, his second one. And then also to John Ritchie that contributed a module that parses the Firefox cache files, although it hasn’t yet been distributed with the tool, sent just slightly too late for me to add it in this release. And I hope that people will continue to contribute modules to the tool.

I know that documentation has been lacking for potential developers. I hope that I will find the time soon to create such documentation, making it easier for people to contribute modules. In the mean time, I suggest looking at previously developed modules and to download the source code and look inside the “dev/” directory. There you can find templates for new modules which can assist in the creation. However better documentation is hopefully on its way

It’s been a while since my last post, and several things have happened since then…. for instance the release of version 0.52 of log2timeline.  I will publish another blog post detailing the difference between version 0.51 and 0.52, such as the use of l2t_process a new tool released alongside log2timeline.

I will also be talking at the upcoming SANS Forensics and Incident Response Summit in Austin, Texas. I’m going to be discussing the release of version 0.60 of log2timeline (stay tuned), which I’m working hard at getting ready (the reason for the few blog posts lately). There will be a lot of great talks at the summit, so I urge you to check it out…

And the nominees for the Forensic4Cast awards has been posted, and for some obscure reason someone decided to nominate log2timeline as the best computer forensics software.  I’m honored that the tool got nominated, something that was really a surprise to me, and given the competition (FTK and Encase) the tool is really up against giants in the field.

In this second post in my short series of timeline analysis I’m going to discuss the use of the CSV output module.  In my previous post I discussed a bit about the different modules there are in log2timeline, at least the version that was released then, and the meaning of each entry within the mactime format.  However since then, I’ve started to use the CSV output module instead of the mactime one, which will be the focus today.

After reading the excellent blog post about reviewing timelines using Excel by Corey few months ago I immediately told my self…that’s exactly what I was planning to do for my second series of the timeline analysis post.  However, I wanted to focus more on the CSV output and the benefits of using that, so I decided to a similar blog post as he did, just with a alternative method.  Somehow this blog post just got buried away somewhere far, far away and never got written… well, until now.  You can read the original post by Corey here, and I’m not going to try to repeat what is said there, please read the post yourselves as it contains lots of useful information that I’m not including here.

The problem I’ve got with the mactime format and Excel is the fact that filtering is not really optimal when it comes to something like the supertimeline.  That is to say, it is easy to filter out dates, but as soon as you start trying to filter the description field you start to hit the limitations of Excel quite quickly, just like Corey pointed out.  However, the CSV output module of log2timeline splits up the information into more fields, allowing greater and more fine grained filtering, making the basic filters more useful and the need to go to advanced filters less likely.  So I’m going to focus on that aspect in this post.  The CSV output consists of the following fields:

date,time,timezone,MACB,source,sourcetype,type,user,host,short,desc,\
version,filename,inode,notes,format,extra

The fields meaning is:

• Date: The date of the event, in the format of MM/DD/YYYY
• Time: The time of day, expressed in a 24h format, HH:MM:SS
• Timezone: the timezone that was used to call the tool with.
• MACB: The MACB meaning of the fields, mostly for compatibility with the mactime format.
• source: The short name for the source.  All web browser history is for instance WEBHIST, registry entries are REG, simple log files are LOG, etc.
• sourcetype: A slightly more comprehensive description of the source, “Internet Explorer” instead of WEBHIST, “NTUSER.DAT” instead of REG, etc.
• type: The type of the timestamp itself, such as “Last Accessed”, “Last Written” or “Last modified”, etc.
• user: The username associated with the entry, if one is available.
• host: The hostname associated with the entry, if one is available.
• short: A short description of the entry, usually contains less text than the full description field.
• desc: The description field, this is where most of the information is stored, the actual parsed description of the entry.
• version: The version number of the timestamp object.
• filename: The filename with the full path of the filename that contained the entry
• inode: The inode number of the file being parsed.
• notes: Some input modules insert additional information in the form of a note, which comes here.  Or it can be used during the review by the investigator.
• format: The name of the input module that was used to parse the file.
• extra: Some additional information parsed is joined together and put here.

Now we only need to create the timeline… the first step is to generate a timeline using TSK, save to a file, let’s call it “bodyfile”. Then to generate the timeline using log2timeline (in this case we are dealing with a Windows XP image):

cd /mnt/analyze
timescanner -d .  -z local -f winxp -o csv -w /cases/123456/timeline.txt

Now you got two files, one being in CSV format the other in mactime (the one produced by TSK).  Now we need to convert the mactime format into CSV, again using log2timeline to do so. We use the mactime input module to read in the bodyfile, and we append it to the timeline.txt file we created earlier.

log2timeline -f mactime -z local -o csv -w timeline.txt bodyfile

This way we have a file, “timeline.txt”, which is a CSV file containing both the timeline extracted using TSK and log2timeline.  We can now open this file up in Excel.  I’m using the Mac OS X version, so the screenshots might differ a bit from the Windows version, yet the principles should all stay the same (Windows screenshots can be found in Corey’s post here).

Import the File Into Excel

The first step obviously is to open Excel and import the file.  That can be done in two ways, either simply opening the file itself or by choosing “File/Import”.

If you decided to go for the “import” function, the next screen is a choice of file type.  Here we will choose a text file instead of the default value of a CSV file.

Choosing the file type in the import menu

The next step of the text import wizard gives us the choice of either splitting the file using a fixed with or delimited.  Since this is a comma separated file, we will choose “Delimited”.

The file is comma delimited, so choose delimited.

The third step of the wizard let’s us choose which delimiters we’ve got.  We only want to use the “comma” option, so please un-check the “Tab” field and check the “comma” one.

Check the box marked "comma", and un-check the box "tab"

The third and final step of the wizard is to choose the data type for each column.  I usually choose the value of the “date” field as a “Date” of the format “MDY” to make things a bit easier in the final step.

In the last window in the text import wizard, choose the date field as the value "Date: MDY"

Sorting

Now all the data is imported, one thing to notice is the fact that timescanner does not sort the timestamps at all.  This is due to the fact that timescanner recursively scans the image, and parses each file it finds, inserting the events as it goes. Therefore we need to sort the data, and create filters.  Start by highlighting the top row and choose “Filter” from the “Data” menu.

Highlight fields and choose "Data/Filter"

This turns all of our fields inside the top row into filterable columns, with a drop-down menu.  Now all we need to do is to sort the data in the correct time order.  So we go for the “Data / Sort…” option in the menu.

Time to sort the data

To correctly sort the data we start by sorting the fields based on the “Date” column, and we usually want to have the latest events on the top, so we choose to sort on value in the order of “Newest to oldest”.  However this is not enough, since we’ve also got the time field.  We therefore add another field using the “+” sign, there we choose the column “time”, which we also sort on values based on the order of “Largest to Smallest”.

Sorting the timestamp based on date from newest to oldest and time on largest to smallest

Now we’ve got our timeline all sorted out and ready to analyse.

Analysis

Just to show the simple filters and what we can do with them.  Now we can easily sort based on the sourcetype for instance.  Just click on the arrow next to the “sourcetype” column and choose which sourcetypes you would like to include in the analysis.

Filtering based on sourcetype

Now you can easily choose which days to examine, months or years.  Simply use the “date” column and filter based on that.  Here we are only focusing on January 2009 (or janúar as it is written in Icelandic).

Filtering dates, simple filter

One final trick I use quite frequently with the simple filters.  As soon as I’ve gone through some timestamps that are of some value, I often highlight them and give them color, usually only use few colors.  Then I can filter out or include only events of certain color.

Fields can be color coded

Filtering based on color:

Finally the CSV output contains often too many columns.  So it is often wise to hide few columns from view.  I usually hide the following fields:

• timezone (the same for everyone usually)
• host (if I’m examining one host only on the timeline)
• short (better to see the full description)
• version (don’t care about that one)
• format

One thing to notice though is the fact that timelines can quickly become way too large for Excel and other spreadsheet application.  The spreadsheet becomes incredibly slow and difficult to manage.  So I usually use some grep and other command-line kung-fu to remove irrelevant entries from the timeline, or only include the timespan of the investigation, that is making it as small as possible.

In my next post, I will go over  some of the command line stuff I usually do to minimize my timeline and other tricks I tend to do before loading it up in Excel.

I will also go over visualization, using the output module of BeeDocs.

And if you have any other suggestions, please add them in the comment section.

One of the things I did to make updating and maintaining log2timeline easier was to create an apt repository so that the tool could be easily installed using apt-get. However I hadn’t done so prior to the release of the SIFT workstation, so SIFT was released using a source installation of the tool.  That also meant that some of the dependencies were installed using CPAN, which Debian/Ubuntu doesn’t always like.  And with the release of version 0.50, there were quite a few libraries that had to be installed.

This really pushed me into creating a short HOWTO describing the process of removing the source installation and migrate to the easier to maintain Ubuntu repository.  That way the tool gets updated along with every OS update.  This HOWTO was supposed to be published a long time ago, but somehow got left behind… so finally here it is…

HOWTO upgrade log2timeline on the SIFT workstation

The first thing that has to be done is to add the log2timeline repository to the sources.list file.  Since I haven’t gone through the steps of releasing log2timeline into the main Ubuntu repository, this approach has to be done (at least for now).

echo "deb http://log2timeline.net/pub karmic main contrib non-free"
>> /etc/apt/sources.list

Since all of the Debian/Ubuntu packages get signed with my PGP key you will need to install it in your apt-key chain, otherwise you will get complaints.  So that is the next step.

cd /tmp
/usr/bin/wget http://log2timeline.net/gpg.asc
/usr/bin/apt-key add gpg.asc
/bin/rm gpg.asc

Now you’ve installed my GPG key and added the repository to the sources file. Now it is possible to update the Ubuntu packages by issuing:

apt-get update

Since we’ve already installed the tool we need to remove the log2timeline libraries and at the same time remove some of the libraries that got installed using CPAN and will now be installed using Ubuntu packages (otherwise the system will complain since there will be two instances installed and registered).

rm -rf /usr/local/share/perl/5.10.0/Log2t/
rm -rf /usr/share/perl5/Log2t/
rm /usr/local/bin/log2timeline
rm /usr/local/bin/timescanner
rm /usr/local/bin/update_log2timeline
rm -rf /usr/local/share/perl/5.10.0/XML
rm -f /usr/local/share/perl/5.10.0/Parse/Evtx.pm
rm -rf /usr/local/share/perl/5.10.0/Parse/Evtx

Now we’ve removed all the CPAN installed libraries that could create conflicts and installed the repository to our apt sources.  So the last step on the way would simply be to install the tool

apt-get install log2timeline-perl

This should be it… and to verify when all the installation is completed

log2timeline -V
log2timeline -f list

Automation

And to make things even easier… I’ve created a simple Bash script to automate this. So the process will be even simpler.  Just download the script from the SIFT workstation and run it

wget http://blog.kiddaland.net/dw/migrate_log2timeline_to_apt.sh
chmod 755 migrate_log2timeline_to_apt.sh
sudo ./migrate_log2timeline_to_apt.sh

And everything should be updated and completed.

Update

I updated the path to the script, that was obviously wrong (gave a 404 error).  It should work now

Well, its been quite a while since my last post, summer vacation coupled with paternity leave gave me a pleasant absence from the computer screen. But I’m back now, and surprisingly my gold paper got finally been published.  The title of the paper is “Mastering the Super Timeline With log2timeline”, and for those that carefully read the title it describes my little pet project of log2timeline and timeline analysis in general.

And I’m about to give a talk at the SANS EU Forensics summit taking place in London on the 8th and 9th of September. Well unless some unnamed volcanoes here in Iceland start to protest again… it will take place then.

Well, I’ve finally decided to release version 0.50 of log2timeline.  Lot of things have changed since version 0.43, although there is only one new input module introduced to the tool, we will get to that later.  I just wanted to go over some of the changes made to the tool.

First of all the verification phase has been slightly changed in all input modules to make it slightly more optimized, thus making timescanner run considerably faster on a mounted image.  Secondly, which is perhaps the biggest structural change, is the modification of the timestamp object as it is called within the framework.  The timestamp object is basically a Perl hash that contains information about the parsed line or timestamp within a file being parsed by the tool.  The timestamp object is created by the input module, and then used by the output module to create the appropriate output.  The old structure of the timestamp object was too dependent on the mactime output, making it difficult to create new output modules, without repeating some information.  So the timestamp object was completely changed, making it independent of any particular output method, making the output modules need to process the data a bit more, but instead make the output a lot more intuitive.

Other very important changes is the addition of code contribution in the tool, for the very first time someone actually contributed code to the tool (something I’m hoping is a new trend, not just an once in the lifetime of the tool).  One new input module for parsing the output of both psscan and psscan2 modules in volatility was added by Julien Touche as well as the new timescanner_threaded application that was contributed by Ben Schmitt.  One of the new things about version 0.50 was supposed to modify timescanner so it would be a threaded application, making it considerably quicker than the older single-threaded one.  With version 0.50 the very first threaded version is released as a proof of concept, and it is not recommended for real use, since it really isn’t any faster than the single-threaded one and sometimes it skips printing some of the timestamps.  So do not use it for anything else than to test it, and report bugs.  I’m hoping that in the next version we will have this fixed, perhaps it needs to be completely rewritten to work properly, perhaps there are just few bugs that we need to sort out.

Another change that has to be mentioned is the ability of timescanner to choose which modules to use.  The older versions had the mentality of either use every input module (timescanner) or just a single one (log2timeline).  Version 0.50 introduces the possibility to manually select which modules are loaded up by timescanner and used for the recursive scan.  The option -f of timescanner is the key here in choosing the appropriate modules.  The option can be used in the following way:

• -f module1,module2,module3
  The first option is just to list all the modules that you want to use, comma separated.
• -f=-module1,module2,module3
  The second method is to use the minus (-) signal to indicate which modules you want to skip.  However it should be noted that if you use the – signal you have to prepend it with an equal sign ( -f=-module1).  This tells timescanner to load up all available input modules EXCEPT the ones that are listed up.
• -f list
  The third option is to use a pre-defined lists of modules to use.  These pre-defined lists are simply text files that contain the names of the modules to use.  The current lists that are included with the tool are:

web
 chrome, firefox3, firefox2, ff_bookmark, opera, iehistory, iis,

winvista
 chrome, evt, exif, ff_bookmark, firefox3, iehistory, iis, mcafee, opera, oxml,
pdf, prefetch, recycler, restore, sol, userassist, win_link, xpfirewall,

winxp
 chrome, evt, exif, ff_bookmark, firefox3, iehistory, iis, mcafee, opera, oxml,
pdf, prefetch, recycler, restore, setupapi, sol, userassist, win_link,
xpfirewall,

There are several other changes that have been made, so read the full changelog to see a list of all changes.  Two new output modules have been added to the tool, TLNX which is simply a TLN output in a simple XML format.  The other new module is the BeeDocs, which is a timeline visualization tool that runs on Mac OS X.  The output module saves a tab delimited text file that can be imported directly into the tool.

I was also supposed to give a talk about log2timeline at the SANS EU forensic summit, that got canceled because of our lovely volcano here in Iceland.  I promised to release my presentation as soon as I would release the new version, so here it is. The presentation contains among other screen shots of the BeeDocs output module as well as some better description of the timestamp object and the inner structure of log2timeline.