Tag: Streaming

Analytics Hub Cloud Pub/Sub Data Analytics Official Blog Streaming July 15, 2024

Share your streaming data with Pub/Sub topics in Analytics Hub - Analytics Hub now supports sharing Pub/Sub topics, enabling organizations to curate, share, and monetize their streaming data assets. By leveraging Analytics Hub Exchanges and Listings, businesses can logically categorize and group sets of Pub/Sub topics and provision access at scale.

Cloud Dataproc Data Analytics Official Blog Streaming July 15, 2024

Deployment patterns for Dataproc Metastore on Google Cloud - This blog post explores four DPMS deployment patterns: a single centralized multi-regional DPMS, centralized metadata federation with per-domain DPMS, decentralized metadata federation with per-domain DPMS, and ephemeral metadata federation. Each pattern has its own advantages and disadvantages, and the best choice for an organization will depend on its specific needs and requirements.

Data Analytics Datastream Official Blog Streaming July 8, 2024

Announcing new stream recovery capabilities for Datastream - Datastream stream recovery enables quick resumption of data replication with minimal to no data loss in scenarios like database failovers or network outages.

Data Analytics Datastream Official Blog Streaming June 24, 2024

Simplify historical data tracking in BigQuery with Datastream's append-only CDC - Datastream's append-only mode simplifies change data capture by preserving every change as a new row in your target BigQuery table. It offers cost efficiency, improved data accuracy, and real-time insights. With append-only mode, businesses can maintain a historical record of changes, track data modifications, and gain deeper insights from their data.

Cloud Dataflow Data Analytics Official Blog Streaming June 10, 2024

Boost developer productivity with new pipeline validation capabilities in Dataflow - Dataflow pipeline validation is now generally available. It performs dozens of checks to ensure that your batch or streaming job is error-free and can run successfully.

BigQuery Cloud Dataflow Official Blog Streaming June 3, 2024

Accelerating CDC insights with Dataflow and BigQuery - This post covers how to use BigQuery’s new CDC capability in Dataflow along with the new Dataflow at-least-once streaming mode to simplify your CDC pipeline and reduce costs.

AWS Cloud Pub/Sub Data Analytics Official Blog Streaming June 3, 2024

Easily stream data from AWS Kinesis to Google Cloud with Pub/Sub import topics - Pub/Sub import topics enable streaming ingestion into BigQuery from external sources, with the first supported external source being Amazon Kinesis Data Streams. Import topics provide a simplified way to ingest data from Amazon Kinesis Data Streams directly into Pub/Sub, reducing the complexity of setting up data pipelines between clouds. Once the connection is established, Amazon Kinesis producers can be gradually migrated to Pub/Sub publishers. Data from Amazon Kinesis Data Streams can be routed to BigQuery using BigQuery subscriptions, and Pub/Sub autoscales to adapt to changes in the Amazon Kinesis data stream.

Cloud Dataflow Data Analytics Official Blog Streaming May 27, 2024

More flexibility for your Dataflow jobs with new controls for latency versus cost - Dataflow Streaming Engine users can now choose between lower peak latency or lower streaming costs for their workloads by adjusting the autoscaling utilization hint value. The autoscaling hint value can be set to a higher or lower value using a Dataflow service option. Dataflow’s autoscaling UI provides insights on when it’s worth adjusting the autoscaling behavior and additional dashboards and metrics to monitor the impact of changes.

Data Analytics Official Blog Streaming May 27, 2024

Google Data Cloud innovations for continuous real-time intelligence - Google Cloud offers innovations for continuous real-time intelligence, enabling organizations to harness real-time analytics and make informed decisions. With Dataflow, BigQuery, and Apache Kafka for BigQuery, enterprises can leverage streaming infrastructure for visibility, predictions, and activation. Customers like Spotify, Puma, Compass, and Tyson Foods have achieved significant business impact using Google Cloud's data, AI, and real-time solutions.

Cloud Dataflow Official Blog Streaming May 20, 2024

No work items left unturned: How Dataflow mitigates stragglers