The IceCube Public Data Project

The purpose of IceCube's open data format is to provide the general scientific community access to IceCube's data, without requiring the use of internal proprietary software. More information about the IceCube project can be found here. For more information and for IceCube neutrino public data samples see this page.

A list of results and details illustrating the function of the IceCube detector can be found on this page.

Project Overview

This project aims to provide the community with calibrated IceCube data, without reconstructions, consisting of the run number, trigger information (i.e. time and type), and an array of photo-electron signals. Photo-electrons represent the electrical signal created when photons strike the surface of IceCube's Digital Optical Modules (DOM), so their amplitude (i.e. "charge"), position (the position of the DOM that was struck), and the photon's arrival time are recorded in the data.

The data will be grouped together by year starting with the 2011 86 string (i.e. complete) detector. The size of each dataset will be on the order of 10 TB (TeraByte) per year and instructions on how to obtain datasets will be provided through this project. Along with the full datasets, smaller neutrino samples and test datasets will also be provided.

Below is a brief list of what this project will provide:

Getting the Data - Instructions on how to obtain full datasets (or smaller test samples) will be provided. The final dataset sizes are projected to be on the order of 10 TB (TeraByte) per year.
Bytestream Description - A low-level description of the data format will be documented, allowing users to write their own readers and analysis tools in the language of their choice (e.g. C/C++, Fortran, IDL, etc...).
Yearly Configuration - Documentation about the yearly detector configuration, including the geometry and trigger conditions will be provided.
Tools - For convenience a lightweight toolset (e.g. reader and event viewer) written in Python can be downloaded to allow users to quickly gain access to the data.

IceCube86/IceTop80 Triggers by Year

Season I May 13, 2011 - May 14, 2012
Season II May15, 2012 - May 1, 2013
Season III May 2, 2013 - ongoing
Large NChannel sample from April 27th 2013
Neutrino enriched sample from January 10th 2013

The IceCube Coordinate System

The position of the hits (x,y,z) are in units of meters where the +y-axis is in the direction of grid north (i.e. parallel to the prime meridian), the +x-axis is in the direction of grid east, and the origin is in the center of the IceCube detector, approximately 1500m below the surface of the ice.

The Event Bytestream

The Event is a python class which contains all the information in an event. The binary file is then simply a stream of serialized Event objects. The documentation for the source code can be found here. Below is the defintion of the event bytestream:

4 bytes - runID(unsigned int)
4 bytes - year(unsigned int)
8 bytes - startTime (long) Number of tenths of nanoseconds since the beginning of the year.
8 bytes - eventLength(float) - Units of microseconds.
4 bytes - (long) - Number of triggers. This is not a member of the Event class, since it's simply the size of the trigger list. For each trigger the byte structure is given as :
1. 8 bytes - trigger time (float) - Time of the trigger with respect to the start of the event.
2. 4 bytes - nchar (int) - Number of characters in the trigger name.
3. nchar * 1 byte - The characters that make up the trigger name.
8 bytes - nhits (long) - Number of hits in the event. The next set consists of nhits*5*8 bytes (one chunk of 8 bytes for each float of q,t,x,y,z).

Getting Started

To get up and running quickly all one needs is access to Python, which is widely distributed on nearly all systems and certainly available on even more. Access to python, however, is not a requirement. This document provides enough information about the data structure that the stream can be decoded in any language (e.g. C/C++, Fortran, IDL, etc...) and on any archictecture (e.g. 32-bit, 64-bit, big and little endian). What this project provides is the outline of the data structure and a set of tools to decode it. The data is provided as a stream of triggered and filtered events. In each event the timestamp and basic hit and trigger information is provided.

Accessing the Data - Here is a link to a very small fraction of the data and a geometry file. In the future this will be replaced with instructions on how to get the full dataset.
Getting the Tools - Check out the ODF tools. These are not strictly needed but will allow quicker access to the data.
Installing Python - Most systems come with a native installation of Python. Make sure the version is 2.6 or greater.
Installing MatPlotLib - To use the provided event viewer you need to install MatPlotLib.
Installing iPython - For better performance from the event viewer it's recommended that you install iPython.
Setting up your environment - To use the tools provided you only need to source the env.sh script provided. To easily follow the examples below unpack the tools first, and then unpack the data in that same directory.
```
      $ tar -zxvf odf_tools.tgz
      $ cd odf-tarball
      $ cp ~/Downloads/test-data.odf.gz .
    
```

Reading the data - Below is an example python session that illustrates how to read the data with a tools provided from within an iPython session.

In [1]: from fileio import read
In [2]: import gzip 
In [3]: odffn = "test-data/Level1_IC59_data_Run00115150_Part00000000.odf.gz"
In [4]: f = gzip.open(odffn)
In [5]: ev = read(f)
In [6]: print ev.runID
115150
In [8]: print ev.event_time()
2010-01-01 16:44:13.1945487298
In [10]: print ev.eventLength
20.0
In [11]: print ev.triggers
[(10000.0, 'IN_ICE_STRING'), (10000.0, 'IN_ICE_SIMPLE_MULTIPLICITY')]
In [12]: print len(ev.hits)
30
In [13]: print ev.hits[0]
(0.27176223088296952, 11293.742776375493, 114.38999938964838, -461.98999023437466, -369.39001464843727)

Tutorials

The event reader is a simple function that uses the provided write in the fileio module to translate IceCube events from the open data format. When the end of the file is reached read returns None. Below is an set of tutorials that show how to read the data.

Last Updated 17 Oct 2024
by E. Blaufuss.