Information security: It takes an ecosystem

The top 10 data breaches of 2015 included government agencies, healthcare organizations, retailers and security software vendors, according to information available through the Privacy Rights Clearinghouse, a non-profit corporation focused on privacy issues. The well-known breach at the federalOffice of Personnel Management (OPM) affected more than 20 million individuals, revealing detailed personal data used in background checks. Even credit service provider Experian was not immune, with applicant information for one of its services hacked, potentially revealing personally identifiable information (PII) such as social security numbers and birth dates….

Leveraging Big Data for Security Analytics

In September, Hortonworks partnered with ManTech and B23 to foster a vibrant open community to accelerate the development of OpenSOC. In December we additionally partnered with Rackspace Managed Security and submitted OpenSOC to the Apache Incubator as a podling under the name of Apache Metron. A decision to rename the project was made to represent the new direction and the new community. Now the process of graduating Metron to a top-level project (TLP) has begun. Given our proven commitment to the Apache Software Foundation process, we feel uniquely qualified to bring this important technology and its capabilities to the broader open source community. Metron integrates a variety of open source big data technologies in order to offer a centralized tool for security monitoring and analysis. Metron provides capabilities for log aggregation, full packet capture indexing, storage, advanced behavioral analytics and data enrichment, while applying the most current threat-intelligence information to security telemetry within a single platform….

Hortonworks, ManTech and B23 Join Forces to Deliver Advanced Cyber Security Solutions powered by OpenSOC

SANTA CLARA, Calif. and NEW YORK, Sept. 29, 2015 /PRNewswire/ — Hortonworks, Inc. (NASDAQ: HDP), together with ManTech and B23, is working to foster a vibrant open community to accelerate the development of OpenSOC, an open source cyber security analytic platform built to rapidly detect and respond to advanced security threats. With the advent of the Internet of Anything, the security of data is paramount. Hortonworks is fully committed to supporting the innovation of the OpenSOC community and further advancing cyber security technology. OpenSOC works in conjunction with Hortonworks Data Platform (HDP™) and Hortonworks DataFlow (HDF™) to provide a real-time enterprise threat detection system that: Delivers actionable insights from real-time and historical network threat alerts Ingests, stores and analyzes over a million network packets per second Integrates into existing enterprise environments…....

Next Generation Cybersecurity Analytics – Part III, Why a Next Generation OpenSOC is Required

This is the final blog in the 3-part blog series Next Generation Security Analytics. You can find Part I and Part II in the B23 Blog section as well. The initial release of OpenSOC was a pioneering event in that it was the first open source, domain-specific, solution-oriented project that demonstrated the use of distributed processing applications like Hadoop, Kafka, Storm, and Elasticsearch. (that was a mouthful!) For most people even in the Big Data community, these solutions only existed inside commercial proprietary software tools, or within an elite few technically adept companies like Yahoo!, Netflix, Amazon, Twitter, and LinkedIn. For a lot of people OpenSOC helped rationalize the real-world use of Hadoop outside of word count! Credit should go to the original founders and to Cisco for sponsoring such an ambitious project and helping a large audience further understand the power of distributed processing systems. In recent months, though, the official sponsorship of OpenSOC has ground to a halt. Viewing the commit history of the project itself, it was apparent to us and our customers that the commitment to OpenSOC as an open source solution was not where it needed to be. Our pull requests were going unheeded, and as of the date of this post the last accepted commit to opensoc-streaming, the big data processing component, was April 4, 2015, almost 5 months ago.   We believe OpenSOC offers tremendous opportunity, and that there currently exists two major areas of improvement required for this to happen. The first is that the OpenSOC initiative requires transparency. The second is that OpenSOC needs a technical facelift to bring it to 2015 and beyond. In the first case, transparency is necessarily for organizations who wish to embrace OpenSOC in more than a hobby-shop manner. There is no release schedule, no roadmap, no technical discussion about incorporating new features, and seemingly no one at the helm to accept pull requests for community software commits. The primary mechanism to communicate is through the Support...

Next Generation Cybersecurity Analytics – Part II, Technical Overview

In our previous post, Next Generation Cybersecurity Analytics we wrote about an OpenSOC implementation project for a financial services firm (“Bank”). In this post we will go into more technical detail on each of the individual components, explain the data flow, and share our results and conclusions. In our final follow-on post we will make the case why a next generation cybersecurity analytics platform is required. Network Packet Capture The out-of-the-box utilities provided by the open source OpenSOC platform could not meet the scale of network packet capture from the Bank’s high performance hardware collection systems. As a result, B23 developed custom packet capture software utilities to address the Bank’s high performance hardware systems. Peak collection capability for the solution equated to capturing approximately 1 Petabyte (“PB”) of data every 66 minutes from the Bank’s private datacenters. As a first step to enabling OpenSOC, the B23 team started work developing a parallelized software solution to keep up with the high throughput demands of full fidelity packet capture. Since the initial 1.2PB cluster was not scoped to handle such high throughout, the B23 team built throttling mechanisms into its custom-built, industrial-scale packet capture software utilities. Apache Kafka The custom network packet capture software asynchronously submitted the raw packet data to Apache Kafka, managed by Apache Ambari and monitored with an open-source Kafka Manager supported by Yahoo! Due to the dynamic nature of network traffic, Kafka provided the scalable, distributed queuing system required to handle peak ebbs and flows of spikey collected network packets. Using the kafka-python library, we were able to submit asynchronous batches of packets using a simple, round-robin producer. As we increased throughput of packets, we observed that increasing the number of Kafka partitions per topic allowed for near linear scalability. Apache Storm Built into the OpenSOC-streaming package is an Apache Storm spout to pull the raw data from Kafka...