The IceCube Public Data Project
The purpose of IceCube's open data format is to provide the general scientific community
access to IceCube's data, without requiring the use of internal proprietary
software. More information about the IceCube project can be found
here. This project addresses the Data Release Set 2 in the IceCube
Open Data Policy. For IceCube/AMANDA neutrino public data samples
see this page.
Below are short list of relevant papers illustrating the function of the IceCube detector.
This project aims to provide the community with calibrated IceCube data, without reconstructions,
consisting of the run number, trigger information (i.e. time and type), and an array of
photo-electron signals. Photo-electrons represent the electrical signal created when photons
strike the surface of IceCube's Digital Optical Modules (DOM), so their amplitude (i.e. "charge"),
position (the position of the DOM that was struck), and the photon's arrival time are recorded
in the data.
The data will be grouped together by year starting with the 2011 86 string (i.e. complete) detector.
The size of each dataset will be on the order of 10 TB (TeraByte) per year and instructions on how
to obtain datasets will be provided through this project. Along with the full datasets, smaller
neutrino samples and test datasets will also be provided.
Below is a brief list of what this project will provide:
- Getting the Data - Instructions on how to obtain full datasets (or smaller test samples)
will be provided. The final dataset sizes are projected to be on the order of 10 TB (TeraByte) per year.
- Bytestream Description - A low-level description of the data format will be documented,
allowing users to write their own readers and analysis tools in the language of their choice
(e.g. C/C++, Fortran, IDL, etc...).
- Yearly Configuration - Documentation about the yearly detector configuration, including
the geometry and trigger conditions will be provided.
- Tools - For convenience a lightweight toolset (e.g. reader and event viewer) written
in Python can be downloaded to allow users to quickly gain access to the data.
IceCube86/IceTop80 Triggers by Year
The IceCube Coordinate System
The position of the hits (x,y,z) are in units of meters where
the +y-axis is in the direction of grid north (i.e. parallel to the prime
meridian), the +x-axis is in the direction of grid east, and the origin
is in the center of the IceCube detector, approximately 1500m below
the surface of the ice.
The Event Bytestream
The Event is a python class which contains all the information in an
event. The binary file is then simply a stream of serialized Event
objects. The documentation for the source code can be found
Below is the defintion of the event bytestream:
- 4 bytes - runID(unsigned int)
- 4 bytes - year(unsigned int)
- 8 bytes - startTime (long) Number of tenths of nanoseconds since
the beginning of the year.
- 8 bytes - eventLength(float) - Units of microseconds.
- 4 bytes - (long) - Number of triggers. This is not a member
of the Event class, since it's simply the size of the trigger list.
For each trigger the byte structure is given as :
- 8 bytes - trigger time (float) - Time of the trigger with respect
to the start of the event.
- 4 bytes - nchar (int) - Number of characters in the trigger name.
- nchar * 1 byte - The characters that make up the trigger name.
- 8 bytes - nhits (long) - Number of hits in the event. The next set
consists of nhits*5*8 bytes (one chunk of 8 bytes for each float of q,t,x,y,z).
To get up and running quickly all one needs is access to Python, which is
widely distributed on nearly all systems and certainly available on
even more. Access to python, however, is not a requirement. This
document provides enough information about the data structure that
the stream can be decoded in any language (e.g. C/C++, Fortran, IDL, etc...)
and on any archictecture (e.g. 32-bit, 64-bit, big and little endian). What
this project provides is the outline of the data structure and a set of
tools to decode it.
The data is provided as a stream of triggered and filtered events. In
each event the timestamp and basic hit and trigger information is provided.
- Accessing the Data - Here
is a link to a very small fraction of the data and a geometry file. In the future this will be replaced with instructions on how to get the full dataset.
- Getting the Tools - Check out the ODF tools. These are not
strictly needed but will allow quicker access to the data.
- Installing Python - Most systems come with a native installation of Python.
Make sure the version is 2.6 or greater.
- Installing MatPlotLib - To use the provided event viewer you need to install MatPlotLib.
- Installing iPython - For better performance from the event viewer it's
recommended that you install iPython.
- Setting up your environment - To use the tools provided you only need to source the env.sh script provided.
To easily follow the examples below unpack the tools first, and then unpack the data in that same directory.
$ tar -zxvf odf_tools.tgz
$ cd odf-tarball
$ cp ~/Downloads/test-data.odf.gz .
- Reading the data - Below is an example python session that illustrates how to read the data with a tools provided
from within an iPython session.
In : from fileio import read
In : import gzip
In : odffn = "test-data/Level1_IC59_data_Run00115150_Part00000000.odf.gz"
In : f = gzip.open(odffn)
In : ev = read(f)
In : print ev.runID
In : print ev.event_time()
In : print ev.eventLength
In : print ev.triggers
[(10000.0, 'IN_ICE_STRING'), (10000.0, 'IN_ICE_SIMPLE_MULTIPLICITY')]
In : print len(ev.hits)
In : print ev.hits
(0.27176223088296952, 11293.742776375493, 114.38999938964838, -461.98999023437466, -369.39001464843727)
The event reader is a simple function that uses the provided write in the fileio module to
translate IceCube events from the open data format. When the end of the file is reached
read returns None. Below is an set of tutorials that show
how to read the data.
Last Updated 12 May 2013
by Alex Olivas.