Why Apache Metron and Cyber Analytics Matters in 2016
Since our last series of blog posts making the case for the Next Generation of Cisco’s OpenSOC solution, we are pleased to announce that the project has been accepted into the Apache Software Foundation as an incubating project to satisfy the vision we set forth (http://bit.ly/1ZO8RD0). This occurred in December 2015 under the stewardship Hortonworks (NASDAQ:HDP).
B23 is now in our second year supporting the open source OpenSOC codebase, and we are proud to represent a significant part of the Project Management Committee (“PMC”) and Committer members within the Apache Metron incubation project.
Over the past two years, we have deployed Metron in a variety of customers and environments each with unique use cases and objectives. As a result of this experience, we have started to develop our own set of best-practices relative to deployment, configuration, and development focus.
B23 is now the longest, continuously operating organization to support this code base. We are flattered that one of our B23 Committers won the vote for the proposed name ‘Metron.’ We will continue to increase our involvement within Apache Metron in the future.
In 2016, we have four (4) focus areas for Apache Metron which are based on our unique set of experiences and capabilities.
1. Enhanced Fidelity — Network Packet Analysis
From a data acquisition perspective, we believe raw packet capture (“PCAP”) is important for understanding the true fidelity of network behavior. Commodity, low-cost storage, and the Hadoop Distributed Filesystem (“HDFS”) make it technically and economically feasible to store, organize, and query enterprise-scale packet metadata and content. While the original OpenSOC project supported PCAP in certain use cases, we have continued to develop features and capabilities to make PCAP collection more central to our Metron deployments. This includes additional temporal and geospatial trending of PCAP information in the Metron operational dashboard in real-time.
2. Bringing Data Science to the Security Operations Center (“SOC”)
Metron has proven its effectiveness and scalability as an aggregator for disparate security information from a multitude of sources. Often its initial value to most of our customers is viewing that security information as events and alerts within a single pane-of-glass. This common operational picture (“COP”) of security information is a great use case in its own right for Metron. Fortunately, we believe there is a lot more potential for Metron.
Apache Zeppelin – Exploratory Analytics
B23 has now incorporated Apache Spark and Apache Zeppelin into our Metron deployments to add a data science and analytical component to cybersecurity. Using network packet data, we are able to ask more complex questions of our data using Apache Spark, and visualizing those results in Apache Zeppelin. OpenSOC had no analytical capability previously, and now with our inclusion of Spark and Zeppelin, it does. Our mission in 2016 is to bring data science to the SOC.
3. Embedded Device Instrumentation using Lower Level Programming Techniques
Embedded devices and the Internet-of-Things (“IoT”) will require a new paradigm for efficiently securing a new generation of distributed and diverse data telemetry sources. Retail, Healthcare, Automotive, and Financial Services will all be impacted by the proliferation of embedded devices.
Metron ships with some basic Python utilities to instrument data from generic devices. These are good for getting started and prototyping. In 2015, we developed several Python probes using advanced multi-threading techniques that could scale up to a certain point based on the underlying hardware. In one case, we “broke” Python after experiencing global interpreter lock when trying to add a network packet to Kafka faster than every 0.0000001 seconds. We realized quickly we needed to adapt. As a result, we started building endpoint collection capabilities using C. The benefits to using C as a low-level programming language are well known. Using C allowed us more granular control and flexibility at a hardware level to accommodate a wider variety of embedded devices, as well as enhancing the speed at which our collection probe could operate relative to Python.
4. Ease of Deployment
One of our favorite quotes that we show customers about the legacy OpenSOC solution came right from an official product description stating “What OpenSOC is not…” in which one bullet point is “…easy to install and get working quickly.” At B23, Necessity is the Mother of Invention. We have developed a capability using well known “devops” automation tools to deploy Metron in minutes versus days and weeks. Our customers don’t want to spend money and time to deploy Metron and its components, but desire to get right to the collection and analysis of network security information. We adhere to this perspective as well, and we will continue to invest time and effort to enhance our automation capabilities to support a variety of different deployment scenarios and environments. Our core work at B23 automating the deployment of large distributed processing systems, as well as massive horizontal scale-out Cloud infrastructures have allowed us to iterate quickly in this focus area.
The Journey Continues in 2016
2016 has started off with an enormous amount of interest and opportunity for Apache Metron. We look forward to continuing this journey by hiring great data scientists and developers to support our focus areas. Most importantly, we will continue to enable more of our customers to operate their business enterprises more securely.
Dave is a co-founder and Managing Partner at B23. Prior to starting B23, Dave was an Account Executive at Amazon Web Services (AWS). Prior to Amazon, Dave has held numerous technology leadership positions at General Dynamics, CA, Gartner, and Northrop Grumman. Over the past 15 years, Dave has developed actionable strategies and successful implementations of key transformative technologies that have allowed customers to gain a competitive advantage in their respective fields. Over the duration of his career, Dave has supported a variety of enterprise customers in the Financial Services, International Telecommunications, and Public Sector markets. Dave started his career as a network engineer at MCI. Dave has an Electrical Engineering degree from the University of Virginia, and an M.B.A. from the Smith School of Business at the University of Maryland at College Park.
#ArtificialIntelligence #BigData #Cybersecurity #DataScience