Coastal Services Center

Data Transport Laboratory (DTL) Projects

---

DTL Project 2: Boeing Data Mart Pilot

Project Overview

The NOAA Coastal Services Center supports the development of Integrated Ocean Observing System related technologies in the local and regional arenas. As part of that effort, the Center worked with the Boeing Company to develop a pilot implementation of a data mart. This project developed an architecture which moves data from local data providers to an aggregation facility and then publishes the data via Open Geospatial Consortium data services.

The system architecture contains five functional components:

• Web scrapers
• Traditional Relational Database Management System (RDBMS)
• A set of functions that create spatial entities in a spatially enabled RDBMS.
• A spatially enabled RDBMS
• OpenGIS Consortium Web Services

The Web scrapers periodically harvest data from Web pages, parse that data, and insert it into the first RDBMS. A set of functions uses the data from the first RDBMS to create spatial entities in the second spatially enabled RDBMS. OpenGIS Consortium Services provide access to those spatial entities.

Project Selection

Suggestions for candidate technologies to be implemented in the DTL are accepted from IOOS community. The primary sources for the identification of candidates are the local observatories, regional associations, the Ocean.us Data Management and Communications team and its expert teams, and deliberate outreach efforts of the DTL staff such as meetings and workshops.

Candidate technologies, nominated through these mechanisms are filtered through a set of criteria to assess their relevance and importance to the stake holders. For more detail on the selection process, see the Selection Process Web page .

The following table lists the scoring and rationale of that scoring for this project.

Criteria	Score	Rationale
Significance to Local Observatories	3	This project explores the difficulties of using traditional methods to harvest data from local providers. This work should educate the selection of technologies for the provision of data at the local level.
Significance to Regional Associations	4	The RAs may well be expected to develop or support the development of regional or thematic data aggregation Centers or Data Marts.
Relevance to DMAC Guidance	4	This project is consistent with DMAC Plan assertions and guidance.
Relative Maturity of Candidate	4	As a pilot project, the components were complete and functional.
Appropriate Level of Effort for DTL	5	The required level of effort to receive the project deliverables and provide public access to them is appropriate.
Coincidence of other Center Efforts	5	This pilot project was created and pursued outside of the DTL. Its adoption by the DTL represents a complete coincidence of the pilot project objectives and the DTL objectives.
Total Score	25

Prerequisites (top)

The data mart components are designed to be installed on two separate servers: one Unix or Linux and one Windows. The Linux server hosts the Web scrapers and data loaders. The Windows server hosts the databases and provides the OGC WFS servies. The Windows server must provide:

• Java SDK 1.4.2_06 or later
• SQL Server with appropriate service packs
• ArcSDE 9.1 with appropriate service packs
• ArcIMS 9.1 with appropriate service packs

The Unix or Linux server must provide:

• Java SDK 1.4.2_06 or later

Software and Documentation (top)

For more information on the data mart architecture and software, please download the software and documentation bundle provided by Boeing. Additionally, an entity relationship diagram of the non-spatial database was created from the installed database instance and is avialable here.