Google Cloud Overhauls Data Products with Streaming Updates

Google Cloud has made significant updates to its data products, including BigQuery, Managed Service for Kafka, and Dataflow. The latest additions aim to reduce operational friction in data systems by providing practical controls for testing, migration, and live processing.

In BigQuery, the new stateful processing option extends continuous queries beyond stateless streaming operations. This feature allows users to apply JOINs, aggregations, and windowing functions within streaming queries, enabling calculations over time windows such as a 30-minute average.

The synthetic data generator for Managed Service for Kafka is now generally available, allowing users to send mock data into a Kafka cluster without modifying an existing client application or creating a separate virtual machine. This tool can be activated in three clicks and begin sending data in less than two minutes.

Dataflow pipeline updates have also expanded, enabling users to stop and replace pipelines, as well as add a parallel pipeline option to move workloads from an old pipeline to a new one with less disruption. A drain timeout setting has also been introduced to limit costs if processing becomes stuck.