SCHEDULE: NOV 12-18, 2011

When viewing the Technical Program schedule, on the far righthand side is a column labeled "PLANNER." Use this planner to build your own schedule. Once you select an event and want to add it to your personal schedule, just click on the calendar icon of your choice (outlook calendar, ical calendar or google calendar) and that event will be stored there. As you select events in this manner, you will have your own schedule to guide you through the week.

You can also create your personal schedule on the SC11 app (Boopsie) on your smartphone. Simply select a session you want to attend and "add" it to your plan. Continue in this manner until you have created your own personal schedule. All your events will appear under "My Event Planner" on your smartphone.

SciHadoop: Array-based Query Processing in Hadoop

SESSION: MapReduce and Network QoS

EVENT TYPE: Paper

TIME: 1:30PM - 2:00PM

AUTHOR(S):Joe Buck, Noah Watkins, Jeff LeFevre, Kleoni Ioannidou, Carlos Maltzahn, Neoklis Polyzotis, Scott Brandt

ROOM:TCC 303

ABSTRACT:
Hadoop has become the de-facto platform for large-scale analysis in commercial applications, and increasingly so in scientific applications. However, applying Hadoop's byte-stream data model causes inefficiencies when used for scientific data that is stored in highly-structured, binary file formats. This limits the scalability of Hadoop applications in science. We introduce SciHadoop, a Hadoop plugin allowing scientists to specify logical queries over array-based data models. SciHadoop executes these queries as map/reduce programs defined over the logical data model. We describe the implementation of a SciHadoop prototype for netCDF data sets, and quantify the performance of three effective optimizations: the first optimization minimizes network traffic by intelligently partitioning the input space of mappers at the logical level; the second optimization avoids full-scans by pruning partitions using knowledge of query data dependencies; the third optimization minimizes data transfers by processing holistic aggregation functions (e.g. median) at mappers instead of reducers whenever possible.

Chair/Author Details:

Joe Buck - University of California, Santa Cruz

Noah Watkins - University of California, Santa Cruz

Jeff LeFevre - University of California, Santa Cruz

Kleoni Ioannidou - University of California, Santa Cruz

Carlos Maltzahn - University of California, Santa Cruz

Neoklis Polyzotis - University of California, Santa Cruz

Scott Brandt - University of California, Santa Cruz

  Click here to download .ics calendar file

  Click here to download .vcs calendar file

  Click here to add event to your Google Calendar

The full paper can be found in the ACM Digital Library