log2timeline updated

March 6th, 2010 kiddi 1 comment

I’ve just released a new version of log2timeline, version 0.42.  The new version includes two new input modules, one for extracting timestamps from PDF metadata and another one from McAfee anti-virus log files.  The new version also includes several bug fixes, the full changelog can be read here. The development focus will be to move the tool to version 0.50, which will introduce a new design of how timestamps and related information is handled within the framework, including a shift to TLN as the standard output format, more details can be found inside the roadmap.

log2timeline will also be included in the upcoming 2.0 release of the SIFT (SANS Investigative Forensic Toolkit) workstation, which will be available soon (and yes it is based on Ubuntu now). That way people can enjoy the tool without needing to go through the installation process with all the needed dependencies.

The agenda for the upcoming SANS EU forensics summit is up. I encourage everyone that has the change to attend this summit, there are some greate talks and of course a great change to meet some of the top experts in computer forensics in Europe.  And of course a change to meet with me and get me to implement some feature to log2timeline that you always wished was there, but for some odd reason you didn’t send me an e-mail to request it.

Small updates

February 17th, 2010 kiddi No comments

Just recently saw a post at Slashdot about Adobe implementing private browsing in their Flash Player.  That means that when the user starts private browsing mode in their web browsers LSO files will not be stored on disk.  This is implemented in the way that during the private browser session all Flash cookies are stored only in memory, and as soon as the browser is closed they are cleared.

Why do we care about this? Well with this change we will start to see that private browsing is becoming more private (or actually private), and it will make our lives as forensic investigators more difficult, since we cannot simply examine Flash cookies to determine the users browsing history (at least partially).

I just posted a blog post in the SANS forensics blog about the structure of LSO files and a quick view of how log2timeline parses it to extract timestamps. I’m not going to repeat that post on this site, so if you would like to know more about the binary format of LSO files, please read the blog post.

Recently there have been a lot of discussions about creating a standard for timeline analysis. Currently log2timeline relies upon the good old mactime format for it’s output (although it is possible to use several different output mechanism), a standard that was created for filesystem timelines.  Although it works great for its original purpose It might not be the most optimal output mechanism when incorporating timestamp information from other sources which is one of the reasons why this push for a new standard has been discussed. The structure of log2timeline will be changed soon to separate the internal structure away from the emphasis on mactime and move to a more neutral approach, and perhaps change the default output mechanism to something like TLN.

With the move to a more neutral approach more logic will be moved into the output modules, meaning that it will be easier to make the description text (which every output module includes) can be more descriptive and does not need to repeat information that might be contained within the output itself (such as TLN which includes a source field, why repeat the source in the description field?)

SANS EU Forensics Summit

January 26th, 2010 kiddi No comments

I just wanted to write a short post about the upcoming SANS European Digital Forensics and and Incident Response Summit that will take place in London on the 19th and 20th of April.  I encourage everyone that has the chance to attend since there are some very interesting talks, such as; Jesse Kornblum’s talk about fuzzy hashing, Keith Foggon’s discussions about trends and techniques and Lee Whitfield’s Windows Shadow Volumes presentation.

I will also be there, talking about log2timeline.  The title of my talk is Mastering the Super Timeline – log2timeline style.  After the talk I will participate in a tool talk panel, so there is your chance to pound me with some difficult questions…

The abstract of my talk is:

Traditional timeline analysis can be extremely useful yet it sometimes misses important events that are stored inside files on the suspect system (log files, OS artifacts).  By solely depending on traditional filesystem timeline the investigator misses context that is necessary to get a complete and accurate description of the events that took place.  To achieve this goal of enlightenment we need to dig deeper and incorporate information found inside artifacts or log files into our timeline analysis and create some sort of super timeline. These artifacts or log files could reside on the suspect system itself or in another device, such as a firewall or a proxy.  This talk will focus on the tool log2timeline, which is a framework built to parse different log files and artifacts to produce a super timeline in an easy automatic fashion, designed to assist investigators in their timeline analysis.

So the talk will contain some of the work in my upcoming Gold paper, titled “Mastering the Super Timeline With log2timeline” (did someone notice the similarities between the titles).  The paper should be published soon, at least before the summit.

Version 0.41 of log2timeline published

January 22nd, 2010 kiddi No comments

I’ve just published version 0.41 of log2timeline, for a full list of the changes read the changelog.  This upgrade is a recommended upgrade since it contains several bug fixes as well as enhancements to the tool.  I’ve added new input modules for: Google’s Chrome History, Opera History, Firefox Bookmarks, and Windows Event Logs (EVTX). I’ve also added a new output module, CEF, for the Common Event Format as designed by ArcSight as well as improving few other input modules (more on that later).

In my last post I talked about Opera history files as well as the bookmark feature of older versions of Firefox.  Since I’ve added support for the bookmarks features in older versions of Firefox (the ones that still store their bookmark information in the bookmarks.html file) I decided to include those information in the newer versions of the browser as well.  As of version 3+ of Firefox it no longer stores bookmark information inside the bookmarks.html file.  Instead it stores them in the places.sqlite database, the same one that contains the browser history.  Therefore I upgraded the firefox3 input module to include information about bookmarks, which are stored inside the moz_bookmarks table as well as in inside the moz_places table.  The SQL command used to pull out information from the bookmarks is the following:

SELECT moz_bookmarks.type,moz_bookmarks.title,moz_bookmarks.dateAdded,
moz_bookmarks.lastModified,moz_places.url,moz_places.title,
moz_places.rev_host,moz_places.visit_count
FROM moz_places, moz_bookmarks
WHERE
 moz_bookmarks.fk = moz_places.id
 AND moz_bookmarks.type <> 3

There is one field in the moz_bookmarks table that is of special interest, that is the “type” field.  There are three different bookmark types:

  • 1 = A bookmark (URL)
  • 2 = A bookmark folder
  • 3 = Separator

The above SQL command returns all values from the moz_bookmarks table (except separators) that have any corresponding fields inside the moz_places table.  This means that the SQL command in fact only returns bookmarked URL’s, not folders.  So another query is made to get the necessary information about bookmark folders:

SELECT moz_bookmarks.title,moz_bookmarks.dateAdded,moz_bookmarks.lastModified
FROM moz_bookmarks
WHERE
 moz_bookmarks.type = 2

This SQL command extracts all the dates associated with the bookmark folders.  But there are other tables within the places.sqlite database that might contain date objects, that is the table moz_items_annos.  This table contains additional information about bookmarks, that is annotations that are made to bookmarks.  The table stores the time when an annotation was added to a bookmark as well as when it was last modified.  The SQL command used to extract this information from the places.sqlite database is:

SELECT moz_items_annos.content, moz_items_annos.dateAdded
,moz_items_annos.lastModified,moz_bookmarks.title,
moz_places.url,moz_places.rev_host
FROM moz_items_annos,moz_bookmarks,moz_places
WHERE
 moz_items_annos.item_id = moz_bookmarks.id
 AND moz_bookmarks.fk = moz_places.id

An example output of the newly upgraded firefox3 input module is the following:

log2timeline -f firefox3 -z local places.sqlite  | grep Bookmark
...
0|[Firefox3] User: smith Bookmark Annotation: [milw0rm exploits and 0day
exploits database] to bookmark [milw0rm] (http://www.milw0rm.com/)|0|0|0|0|
0|1195573631|1195573631|1195573631|1195573631
...
0|[Firefox3] User: smith Bookmark Folder [Bookmarks Menu]|0|0|0|0|0|
1218738203|1218738203|1195573631|1195573631
...
0|[Firefox3] User: smith  Bookmark URL SANS London 2008 (http://www.sans.org/london08)
[london08] count 0|0|0|0|0|0|1218784170|1218784170|1218784170|1218784170

I’ve also upgraded the flash cookie or Local Shared Object (sol) input module considerably.  The older version was not built to support many of the common flash cookies out there so the new version should implement a parser for every known type of objects there are.  Although I’ve seen some flash cookies that the input module is not capable of parsing that have considerably different binary structure. These files might be an older version of the standard and the current version of the sol input module is unable to parse them (and so are every other SOL editors/parsers that I’ve seen).  I will not include any information about the structure of flash cookies in this post, it will be reserved for a later post.

I’ve also decided not to create all my blog posts on this site and then re-post some of them on the SANS forensics blog. Instead I will post some of the blog posts solely on the SANS blog while others will only be here.  On that spirit I wrote a post about Google’s Chrome browser which can be read here. My blog post about flash cookies will also be posted on the SANS forensics blog site.

Updates to log2timeline

January 5th, 2010 kiddi 2 comments

I’ve been working on a new version of log2timeline, which according to the roadmap is a “web history add-on”.  I started by creating an input module to parse the simple format of Opera browser.  Opera browser maintains two main history files, the “Opera Global History” and the “Opera Direct History”, which are both in a plain text format (although different).  Besides these two files there are some timestamp information that can be gathered from the binary file download.dat as well as few other binary files (I’ve already started to create an input module to parse the binary format).

The main history file of Opera is called the “Opera Global History” and it is stored using a plain text file where each visit is logged in four lines of the files, with the following structure:

Title of the web site (as displayed in the title bar)
The URL of the visited site
Time of visit (in Epoch time format)
An integer, representing the popularity of the web site

The other history file is called “Opera Direct History”, which is a XML file that stores typed history (urls that are typed into the browser). The structure of the file is the following:

<?xml version="1.0" encoding="ENCODING"?>
<typed_history>
 <typed_history_item content="URL TYPED IN" type="text"
                  last_typed="DATE"/>
</typed_history>

The input module called opera is able to determine which history file is provided as an input to log2timeline and parse the file accordingly.

An example usage is the following:

log2timeline -z local -f opera Opera\ Global\ History
Local timezone is: Atlantic/Reykjavik (GMT)
Start processing file/dir [Opera Global History] ...
Loading output file: mactime
Starting to parse file using format: [opera]
0|[Opera] User unkown visited http://www.opera.com/portal/startup/ (http:// \
www.opera.com/portal/startup/) [2419292]|0|0|0|0|0|1262716353|1262716353| \
1262716353|1262716353
0|[Opera] User unkown visited http://mbl.is/mm/frettir/ (mbl.is - Fréttir) \
 [-1]|0|0|0|0|0|1262716353|1262716353|1262716353|1262716353
0|[Opera] User unkown visited http://mbl.is/mm/vidskipti/frettir/2010/01/05/ \
fitch_laekkar_lanshaefismat/ (Fitch lækkar lánshæfismat - mbl.is) [-1]|0|0|0 \
|0|0|1262716386|1262716386|1262716386|1262716386

And for the direct history file

log2timeline -z local -f opera Opera\ Direct\ History
Local timezone is: Atlantic/Reykjavik (GMT)
Start processing file/dir [Opera Direct History] ...
Loading output file: mactime
Starting to parse file using format: [opera]
0|[Opera] User unkown typed the URL mbl.is directly into the browser \
(type "text")|0|0|0|0|0|1262716352|1262716352|1262716352|1262716352

I’ve also added a support for Google’s Chrome.  In short Google Chrome stores it’s data in a SQLite database, not unlike Firefox (as of version 3), so creating an input module was quite quick for that browser.  The first version of the input module parses three tables;

  • urls – Contains information about each visited URL
  • visits – Contains the timestamp information from each URL
  • downloads – Contains information about the downloads

Going through the Google Chrome’s setup is something that I will reserver for a future blog post, so an example is given of the usage.  This first version of the input module gathers basic information from the history file and displays it in a timeline, future versions will include more detailed versions (need to do more research to determine some parts of the history format)

log2timeline -f chrome -z local History
Local timezone is: Atlantic/Reykjavik (GMT)
Start processing file/dir [History] ...
Loading output file: mactime
Starting to parse file using format: [chrome]
0|[Chrome] URL visited: http://tools.google.com/chrome/intl/en/welcome.html \
(Get started with Google Chrome) [count: 1] Host: tools.google.com (URL not \
typed directly)|0|0|0|0|0|1261044829|1261044829|1261044829|1261044829
0|[Chrome] URL visited: http://www.google.com/ (Google) [count: 1] Host: www.\
google.com (URL not typed directly)|0|0|0|0|0|1261044829|1261044829|1261044829|\
1261044829
0|[Chrome] URL visited: http://www.google.is/ (Google) [count: 1] Host: www.\
google.is visited from: http://www.google.com/ (URL not typed directly)|0|0|0\
|0|0|1261044829|1261044829|1261044829|1261044829
0|[Chrome] URL visited: http://www.google.is/search?hl=is&source=hp&q=try+a+\
single+google+searcg&btnG=Google+leit&lr= (try a single google searcg - Google\
 leit) [count: 1] Host: www.google.is visited from: http://www.google.is/ (URL \
not typed directly)|0|0|0|0|0|1261044876|1261044876|1261044876|1261044876

Then today I saw a post by H. Carvey about browser forensics where he talked about the bookmark file contained in Firefox’s profile folder. He discussed the bookmark file and the fact that there are fields within it with timestamps, most notably the ADD_DATE and LAST_MODIFIED entries for folders, and ADD_DATE and LAST_VISIT entries for the URLs.  So I decided to create a new input module to parse the file, ff_bookmark, which uses HTML::Parser to parse the HTML document and extract the timestamps that are contained within it.  Sample usage is

log2timeline -f ff_bookmark -z local bookmarks.html
Local timezone is: Atlantic/Reykjavik (GMT)
Start processing file/dir [bookmarks.html] ...
Loading output file: mactime
Starting to parse file using format: [ff_bookmark]
0|[Firefox Bookmarks] User modified the bookmark file|0|0|0|0|0|1198266703|\
1198266703|1198266703|1198266703
0|[Firefox Bookmarks] User modified the bookmark folder [Bookmarks Toolbar \
Folder]|0|0|0|0|0|1194274986|1194274986|1194274986|1194274986
0|[Firefox Bookmarks] User created the bookmark Orðabanki [http://herdubreid\
.rhi.hi.is:1026/wordbank/search]|0|0|0|0|0|1189521200|1189521200|1189521200|\
1189521200
0|[Firefox Bookmarks] User visited the the bookmark [Orðabanki]|0|0|0|0|0|\
1195489127|1195489127|1195489127|1195489127
0|[Firefox Bookmarks] User created the bookmark SANS Institute - \\
International Training Events [http://feeds.feedburner.com/SansInstituteInter\
nationalEvents]|0|0|0|0|0|1193489569|1193489569|1193489569|1193489569
0|[Firefox Bookmarks] User created the bookmark folder [Öryggisvitund]|0|0|0|\
0|0|1191448948|1191448948|1191448948|1191448948
0|[Firefox Bookmarks] User modified the bookmark folder [Öryggisvitund]|0|0\
|0|0|0|1193819896|1193819896|1193819896|1193819896
0|[Firefox Bookmarks] User created the bookmark ISC: Tip #1 [http://isc.sans.\
org/diary.html?storyid=3438]|0|0|0|0|0|1191448977|1191448977|1191448977|1191448977

These input modules are not part of the published log2timeline tool, but they are all available in the development version of the tool, which can be found here.

Categories: Forensics Tags:

Finally a new version of log2timeline

November 25th, 2009 kiddi No comments

I’ve been working on a new version for log2timeline for a while now, and I finally managed to complete some testing on the new code.  There are some significant changes to the way that log2timeline works in the new version, 0.40. Some of them are:

  • All timestamps are now normalized to UTC
  • The GUI, glog2timeline, has been updated so that it is feature compatible with log2timeline’s CLI front-end.
  • timescanner has been further developed so that it now can parse all the artifacts that log2timeline is capable of.

Full list of the changes can be seen in the changelog.

The reason why the timestamps have now been normalized is the fact that some timestamps are stored as UTC while others use the operating system’s timezone settings.  This might not be such a big problem when using the log2timeline CLI, since it only takes one file at a time and produces a body file.  However the investigator had to knew that this particular file was either stored in the local timezone or in UTC.

The real problem arises with the use of timescanner.  When timescanner is used a directory is recursively searched for all parsable artifacts.  This means that the tool parses both the artifacts that have timestamps in UTC as well as those stored in the native timezone settings, and stores them in the same body file.  This causes the timestamps to vary and causes problems during analysis.  For this reason all timestamps are now normalized so that the output is in UTC.  This means however that the investigator now needs to provide the timezone settings of the suspect machine both to the timescanner/glog2timeline/log2timeline tool as well as mactime (if that tool is used to convert the body file to a working timeline).

The new version of the tool has been tested only on a few test cases, so if you find any bugs in it or have comments, don’t hesitate to contact me, kristinn

-->