Modernizing Splunk Platforms with VAST Data

Modernizing Splunk platforms with VAST Data Universal Storage is one way to increase Splunk storage efficiency without substantially affecting average search run time or indexer ingest rate.

Splunk Enterprise is a software platform widely used for monitoring, searching, analyzing, and visualizing data. It captures, indexes, and correlates massive amounts of data in a searchable container and produces graphs, alerts, dashboards, and visualizations. In early 2018, Intel IT built a Cyber Intelligence Platform (CIP) based on Splunk and Apache Kafka. This platform ingests data from hundreds of sources and security tools, providing context-rich visibility and a common work surface, decreasing the time required to identify and respond to sophisticated cyber threats.

Often, Splunk is deployed on high-performance, converged infrastructure. This approach can be costly if the Splunk indexers (servers) are designed with SSDs for storing both “hot” and “cold” data. A converged infrastructure can also lead to the addition of Splunk indexers simply to increase data storage capacity. An obvious alternative is to disaggregate compute from storage. However, disaggregated infrastructure can cause negative performance impacts to both Splunk search run times and data ingest rates.

VAST Data Universal Storage offers a unique solution for disaggregating Splunk platforms. VAST Data uses advanced data reduction algorithms and Intel® Optane™ SSDs to reduce Splunk data storage requirements without sacrificing performance. Intel IT tested a VAST Data all-flash storage enclosure with Intel IT’s high cardinality, production data and reduced Splunk data size by 2.5x. Average Splunk search run time degraded by only three percent, while the Splunk indexers showed a mere 10 percent reduction in data ingest rate. VAST Data Universal Storage can reduce Splunk cold storage capacity requirements and enable independent compute and storage scaling.

Our tests demonstrated the business benefits of modernizing Splunk platforms with VAST Data:

  • Reduce data storage needs. VAST Data Universal Storage can significantly reduce storage needs and organization’s storage costs, with minimal impact to search run time and indexer data ingest rates.
  • Improve operational efficiency. Petabytes of data can reside in less than half of a data center rack.
  • Scale compute and storage independently. This allows organizations to accommodate massive storage growth demands without additional compute infrastructure.

Intel IT is always looking for new capabilities to effectively manage our data platforms while minimizing our total cost of ownership. We need agility to easily add new users, while adding and removing applications. We also need the ability to scale to support new data sources, as well as increased data—for both real-time and long-term use cases. Intel’s Cyber Intelligence Platform (CIP), based on Splunk and Kafka, is one of these evolving platforms. Our proof of concept has shown that the VAST Data Universal Storage solution can help IT achieve hard-drive-class storage economics without sacrificing performance.