New US survey from Unravel and Sapio Research shows big data deployments moving to cloud and Spark adoption rising rapidly

Today, Unravel revealed the results of a big data survey of more than 300 US IT decision makers. There are a lot of interesting findings in the data. Overall, the survey results show that big data is increasingly being deployed in the cloud and that Spark has gained rapid popularity. The data also reveals that enterprises are most concerned with a lack of sufficient technical talent to properly manage big data as well as the perceived high cost of deploying big data.

Overall, the survey data reveals that enterprises are overwhelmingly embracing the cloud, with big data being a major driver in that. According to the survey, 79 percent of the respondents have big data workloads currently running in the cloud, while 83 percent have a strategy to move existing big data applications into the cloud. As I wrote in Forbes recently, the cloud is an ideal place to deploy big data due to its elastic compute requirements. For example, e-commerce retailers and the spiking traffic they experience on a day like Cyber Monday. With its apps in the cloud, they have instant access to additional compute and storage resources to accommodate traffic surges and to scale back down during quieter times. In a traditional on-premises setting, they would have to provision extra servers and other infrastructure that would go unused most of the year.

The survey also revealed that most respondents are embracing both a multi-cloud strategy (56 percent) and a hybrid cloud strategy (56 percent). Hybrid cloud deployments allow enterprises to keep sensitive workloads on-premises, while multi-cloud approaches allow them to cost-optimize between the various platforms.

Other insights include:

• The survey data illustrated that Spark has ascended and may soon eclipse Hadoop in popularity. Spark was the second most deployed big data technology, with 31 percent of respondents deploying it, while 32 percent deployed Hadoop. Spark was the number one big data technology that IT decision makers plan to deploy for the first time in 2019 (16 percent). Spark is frequently used for streaming and analytical applications common in healthcare, e-commerce, social media and financial sectors. Emerging technologies such as IoT also increasingly use streaming apps that rely on Spark.
• Respondents cited a lack of technical talent to manage deployments as their top pain point (45 percent) related to big data. Big data technologies can be complex and often require Ph.D.-level expertise to manage and troubleshoot manually. Unravel’s Data Operations platform eliminates this skill gap by leveraging AI and machine learning to provide automated recommendations and actions to fix bottlenecks and errors in complex data pipelines, turning existing teams into big data experts.
• The second biggest pain point was cost (41 percent). The cloud and a robust approach to Data Operations and performance management reduces the cost per application, cost per query and optimizes overall operating and capital costs by fully optimizing and providing predictability to the end-to-end data infrastructure resources
• Over half of all IT decision makers (57 percent) say big data projects will account for 25 percent or more of their IT budgets in 2019. Overall, 61 percent expect their IT budgets to grow in 2019.
• A large majority (73 percent) of IT pros expect their big data stack to power fully reliable, useful, profitable business applications by the end of 2019.
• The survey also looked at how organizations are using big data. Respondents made it clear that they value big data mostly for security. Fraud detection was listed as the most effective use case for big data, while cybersecurity intelligence was third. Despite that value, respondents also indicated that security analytics was the application they struggled most to get right, illustrating the complexity of managing the real-time streaming data common in these apps.

As we see consistently, enterprises have embraced a wide range of technologies to help them harness the power of data and those investments are starting to pay off, accelerated in part by the availability and maturity of public cloud and hybrid cloud offerings. However, operational challenges remain and these are now being addressed though a more robust Data Operations approach to people, process and technology needs to support production workloads and applications.

Download the full report.

The survey was conducted by Sapio Research on behalf of Unravel in January 2019 Respondents were IT decision makers from firms with more than 1,000 employees. 35 percent of respondents held C-level or VP positions, while the remaining 65 percent were directors or managers.