The Mitre Corporation® is a not-for-profit company that operates multiple federally funded research and development centers (FFRDCs). One of these FFRDCs is the Center for Advanced Aviation System Development (CAASD), which serves the public interest by advancing the safety, security, effectiveness, and efficiency of aerospace in the United States and around the world.
CAASD has been working with the US Federal Aviation Administration (FAA) since 1958 to provide the safest, most efficient aviation system in the world. CAASD provides the FAA with advanced technical capabilities in systems engineering, operational analysis, safety analysis, and prototyping coupled with in-depth domain knowledge in air traffic management and airspace user operations relevant to the National Airspace System (NAS).
Challenge
CAASD offers specialized data analytics, simulation and computer modeling capabilities to evaluate a broad spectrum of NAS systems and
operations. CAASD researches the NAS’ complex system of systems to provide data-informed answers to fundamental questions that the FAA asks continuously, including:
• How can the NAS be more efficient?
• How can the NAS be safer?
Answering those efficiency and safety questions requires that the CAASD team re-create historical operational scenarios to detect and analyze key efficiency and safety events, such as significant traffic bottlenecks and aircraft operations that are outside normal flight procedures. The CAASD team knew that the quality of its analysis would be directly correlated to the amount, breadth, and quality of the source data that it could analyze.
As part of FAA safety data-sharing initiatives, CAASD relied on airline flight data to aid in risk analysis for joint FAA-Industry risk mitigation. From the beginning, it was apparent that strong data governance and security would be paramount to reassuring carriers that their data would be stripped of any data elements that could tie it back to the particular airline that provided it.
Solution
CAASD continuously ingests, stores and analyzes massive amounts of detailed flight data from a variety of sources and enriches it with other data, such as:
• Pilot and air traffic controller voice recordings
• Weather data
• Terrain maps
• Air traffic management system data, and
• Aircraft schedules and flight plans
Altogether, CAASD has over a petabyte of data in its Hortonworks Data Platform (HDP®) clusters, deployed both on premises and in the cloud. The CAASD team uses the system to look at all the flight data holistically and enrich it with additional external data.
The CAASD team migrated its existing Hadoop cluster onto Hortonworks Data Platform for two main reasons. It wanted to avoid vendor lock-in, and also wanted to get as close to the core technology as possible, so that it would be able to adopt new platform innovation in response to its changing needs.
Results
With more than a petabyte of data in its HDP data lake, CAASD has created derived data products. With flight data, combined with data from hundreds of different surveillance sensors, the team can now create logical “flight stories” for any growing number of specific flights. The derived data products are continually stored to enable historical analyses; the archive of flight stories currently spans more than five years.
All of this visibility helps the FAA to identify systematic risks across the NAS and develop mitigation measures with airspace users, such as controller training improvements or changes to operational procedures.