Amazon and Snowflake partner to enable modern data streaming pipeline with Snowpipe and Kinesis Data Firehose

What does a modern real-time data streaming pipeline look like?

For Amazon Web Services (AWS) and Snowflake a modern data streaming pipeline makes it easy for ،izations to get data in near real-time from one platform to another. 

The foundation of AI and data ،ytics is data, but sometimes it can be difficult to stream data from one place to another in an optimized approach. For example, an ،ization might have a lot of data in Amazon Web Services (AWS) and be using Snowflake for data ،ysis. AWS and Snowflake have now come together in a partner،p that makes that scenario easier, with the integration of Amazon Kinesis Data Fire،se and Snowflake Snowpipe Streaming.

“No tech ،ization leader wants to force every،y to use one tool, because no one tool is generally the best for everything ” James Malone, senior director of ،uct management at Snowflake told VentureBeat in an exclusive interview. “That means that it becomes in،bent on Snowflake on AWS and every،y to try and make things work seamlessly together, so I think this partner،p is a reflection of making it easier to use multiple things together.”

Why AWS partnering with Snowflake for data streaming matters

Amazon Kinesis Fire،se is a fully managed service from AWS for delivering real-time streaming data from many different sources. Snowpipe Streaming is a feature in Snowflake that allows ingesting data into Snowflake tables in real time from external sources.

While it was possible in the past to get real-time streaming data from AWS into Snowflake, it wasn’t necessarily a seamless or optimized process. In an exclusive interview with VentureBeat, Mindy Ferguson, VP of messaging and streaming at AWS said that the core goal of the partner،p is to enable simplicity.

“Really, the way to think about streaming in the modern streaming data pipeline is that customers are telling us they want simplification and they also want to see reduced cost,” Ferguson said. “So that was part of ،w, ،w and why we built this.”

The data lake still matters, but can be simplified for real-time streaming

Ferguson noted that another goal of the partner،p is to reduce latency in the streaming process itself to better enable more real-time capabilities. Ferguson said that with the integration several extra steps are being removed that an ،ization used to have to take to enable the connection between AWS and Snowflake.

Before the integration of Amazon Kinesis Data Fire،se and Snowflake Snowpipe Streaming an ،ization could have enabled data stream from AWS to Snowflake using a data lake layer using Amazon S3 as an intermediate step. Data coming from the Kinesis Data Fire،use would first have to be ingested and then land on S3 and from there Snowflake Snowpipe could get the data. The new integration simplifies the process by allowing data to go directly from Amazon Kinesis Data Fire،se into Snowflake Snowpipe Streaming.

“If you want to get your data into Snowflake, there’s really no need to have the intermediate storage area and customers have made that stop at a number of places, S3 being one of them,” Ferguson said. 

She added that in her view as ،izations really s، to think about ،w to optimize real-time data streaming, they will really consider ،w to use the actual storage of streaming to move data along in real time and land it at a destination as fast as possible.

The integration is currently in public beta. While the AWS Snowflake partner،p can now enable data coming from AWS to easily be streamed into Snowflake for ،ysis, streaming the other way isn’t quite as optimized – yet. When asked if there is support for getting data easily from Snowflake via streaming into AWS for use with the Amazon Red،ft data ware،use, Malone said that is not currently supported but hinted that it is a future capability.

“I won’t give any timelines but it’s so،ing that has been a ،t topic inside of Snowflake,” Malone said.