The trend of applying machine learning and artificial intelligence to the mission of cyber defense is one of the most promising activities in the cybersecurity community. The trend towards eliminating data stovepipes to allow analysts to work over all relevant security data is also a very positive movement. Both of those trends are apparent in the new Apache Spot project.
Apache Spot is a community-driven cybersecurity project undergoing incubation at the Apache Software Foundation (ASF). The project is based on Cloudera's big data platform on Intel hardware, leveraging Apache Hadoop for infinite log management and data storage, Apache Spark for machine learning and near real-time anomaly detection, a suite of packaged analytics that provide tools of immediate use to any security ops team (including visualizations, analytics and machine learning tools). This is all integrated together with other tools in a way that just works. It includes a well developed data model for all relevant threat, technology and incident data. And, since it is an open platform, this model is tailorable for any unique needs.
I've been tracking Apache Spot for quite a while. It had its roots in an Intel project and has had great leadership and contributions from Cloudera as well as some of the greatest names in cybersecurity technology. But was very pleased to be able to get a personal demo from Cloudera's director of cybersecurity strategy Sam Heywood during the RSA conference. There is nothing like seeing it in action and clicking buttons myself (a photo from my demo is here):
There is also a growing application ecosystem for sharing advanced capabilities with the community. And Cloudera has just announced support for Apache Spot, so any enterprise who uses it can opt to have commercial grade services and support.
This is a great capability that pulls together all the relevant data that any SOC would want/need for just about every conceivable cybersecurity use case. It can be used for analysis before breach to continue improvements, can be used to assess the nature of threats that might be exploring and attempting breach, and, during or after the attack can be used to rapidly assess what is going on. Since it is based on an open data model and since great thought has already gone into most use cases it is totally extensible to just about any data source and easily tailorable to any need.
Beyond the SOC, Apache Spot will have use cases for compliance teams, hunt teams, and any other specialized function that needs intuitive access to analytical tools over all relevant security data.
If you have any market survey underway for any SIEM tool, log management tool, insider threat capability, forensic tool or other security data capability I would most strongly recommend you check out Apache Spot before making any decision. Spot provides a single consolidated platform for security data that you can put at the center of all your security operation. This is a great way to address the issue of fragmentation and stovepiping of security data that exists in all enterprises today. And, since it works with Apache Spark it comes with fantastic machine learning and artificial intelligence capabilities out of the box and since it also comes with an apps marketplace those and other solutions will only grow.
I would also recommend an in-person demo. Till you can arrange that, the next best option is to see the video below:
For more see: Cloudera.com
Latest posts by Bob Gourley (see all)
- RiskIQ: Tools to Improve Cyber- Situational Understanding in DoD - April 9, 2017
- Learn The Latest On All Things Data At The 25 April 2017 Cloudera Government Forum - April 1, 2017
- Chances to Speak at O’Reilly Media’s Upcoming Conferences - March 24, 2017