Tag: BigQuery
Resources about BigQuery collected from the weekly GCP Weekly newsletter — a hand-curated collection of Google Cloud Platform news, technical articles, tutorials, videos and release notes. Each entry below links back to the original GCP Weekly issue and retains its publish date and title, giving you a focused, searchable knowledge base for this topic.
BigQuery dbt Paywall June 16, 2025How to Build Metadata-Driven Data Pipelines in BigQuery (+ dbt Examples) - Automate, validate, and orchestrate your BigQuery workflows with metadata logic — and make your data platform think for itself.
BigQuery Data Analytics June 16, 2025How to preview table data at no cost in Google BigQuery - The article discusses cost-effective ways to preview table data in Google BigQuery.
BigQuery Migration June 9, 2025Things I wish I knew before migrating Snowflake to Bigquery - Migrating from Snowflake to BigQuery requires careful planning due to key differences like shorter time travel, different CDC approaches, and case sensitivity. Optimization strategies also vary, with BigQuery needing manual partitioning and clustering for peak performance. The author advises early stakeholder engagement, phased migration, rigorous data validation, and continuous cost monitoring to ensure a smooth transition.
BigQuery Terraform June 9, 2025If Only Terraform Used Maps: Fixing BigQuery Schema State Pain - Terraform, while excellent for infrastructure as code, struggles with BigQuery table schemas due to its naive linear comparison strategy. This can lead to unintended column deletion and recreation, potentially causing data loss. The author suggests Terraform should treat the schema like a map instead of an array list to resolve this issue.
BigQuery FinOps June 9, 2025Computing Actual Physical vs. Logical Storage Costs in BigQuery - How to identify immediate potential cost savings, by determining the optimal storage billing model for each BigQuery dataset.
BigQuery dbt Paywall June 9, 2025dbt + BigQuery in Production: 13 Technical Practices to Scale and Optimize Your Data Platform - Best practices using dbt with BigQuery.
BigQuery Data Analytics Official Blog June 9, 2025From analytics to data management: New BigQuery transactional features - New BigQuery features support near-real-time, transactional-style operations, letting you perform common data management in your data warehouse.
BigQuery Data Analytics Official Blog June 9, 2025Maximize BigQuery performance with enhanced workload management - Updates to BigQuery workload management help optimize workloads and resource allocation, preventing performance issues and resource contention.
BigQuery Google Analytics June 2, 2025Analyzing Google Analytics 4 Storage Cost in BigQuery - Metadata analysis deployed as a reusable, open-source function, to give immediate insight into this increasing cost on your cloud bill.
BigQuery Dataform June 2, 2025How Dataform Handles Incrementality in BigQuery - In this deep dive we take a look at the SQL code Dataform generates and what it does under the hood.
AI BigQuery June 2, 2025Reverse ETL in real time with BigQuery using only SQL - This article explains how to implement reverse ETL in real time using BigQuery continuous queries and SQL. It walks through creating reservations, datasets, tables, ML models, and destination tables within BigQuery.
BigLake BigQuery Data Analytics Official Blog Streaming June 2, 2025Google Cloud’s open lakehouse: Architected for AI, open data, and unrivaled performance - New in Google Cloud’s lakehouse are BigLake Iceberg native storage; united operational and analytical engines; and faster BigQuery SQL, to name a few.
BigQuery FinOps Gemini May 26, 2025Analyze Gemini Code Assist usage with BigQuery - Analyze AI coding assistance adoption and impact for your team with Gemini Code Assist metadata logs and BigQuery.
BigQuery FinOps Kubernetes May 26, 2025Optimizing GKE Cost Visibility with Kubecost and BigQuery Integration - The article provides a guide on mastering GKE cost analysis using Kubecost and GCP billing data exported to BigQuery. It outlines how to enable cost allocation for GKE clusters, set up billing export to BigQuery, install and configure Kubecost, and integrate GCP for out-of-cluster resource attribution, enabling detailed Kubernetes spend monitoring and optimization.
ADK BigQuery Vertex AI May 26, 2025Agent Development Kit — NL2SQL with Bigquery - The article introduces the Agent Development Kit (ADK), an open-source framework for developing and deploying AI Agents, and demonstrates its use in implementing NL2SQL with Bigquery.
BigQuery Data Science Python May 26, 2025Generate Structured Output with Gemini in BigFrames - Generate Structured Output with Gemini in BigFrames, an open source Python library offered by Google.
BigQuery May 26, 2025Data to Value Tools - An Open-Source Google BigQuery Toolkit for Analytics Engineers.
BigQuery Data Analytics Official Blog May 26, 2025Introducing AI.GENERATE_TABLE: creating structured data from gen AI models in BigQuery - AI.GENERATE_TABLE() in BigQuery lets you convert unstructured data into a structured table based on the provided prompt and table schema.
BigQuery Data Analytics Official Blog Streaming May 26, 2025Fast, approximate analytics at scale: Apache DataSketches available in BigQuery - Apache DataSketches in BigQuery enable approximate analytics with minimal memory or computational overhead, and with a single pass through the data.
BigQuery FinOps May 19, 2025Your guide to finding inactive tables in Biguery - This article provides a BigQuery script to identify inactive tables and reduce storage costs. The script analyzes `INFORMATION_SCHEMA` views to determine the last usage time of tables based on queries, data changes, loads, and copies. By identifying and removing these tables, users can optimize their BigQuery storage and potentially lower their Google Cloud costs.
BigQuery Dataform Google Analytics May 19, 2025How I Optimized 70TB of GA4 Data in BigQuery Using Dataform - One way to handle when execution time becomes the bottleneck of a data pipeline with Dataform.
BigQuery Cloud Spanner May 19, 2025Spanner & BigQuery — Comprehending Reverse-ETL and its associated trade-offs - The article explores using Reverse ETL to export data from BigQuery to Spanner, highlighting the trade-offs between write throughput and Spanner performance. Testing revealed that high write concurrency from Reverse ETL jobs can significantly impact Spanner read latency and throughput and that having secondary indexes over the target table can cause writes throttling affecting RETL jobs duration. The author provides mitigation strategies like limiting job concurrency, using stale reads, and adjusting BigQuery slots reservation and Spanner nodes provisioning to balance performance and cost.
BigQuery BigQueryML Data Analytics Official Blog May 19, 2025Understand why your metrics moved with contribution analysis in BigQuery ML, now GA - BigQuery ML contribution analysis models, now GA, automate insight generation and output key change drivers from multidimensional data.
BigQuery Billing May 12, 2025BigQuery dynamic reservation: the new query cost hack! - BigQuery cost model and reservation was binary and static. A new features makes them dynamic and opens new possibilities!
Airflow BigQuery Docker May 12, 2025Fix: “Compute Engine Metadata server unavailable” in Airflow Docker for GCP Authentication - Running Airflow in Docker locally? Trying to reach BigQuery and getting errors? You’re not alone. Here’s how to solve it.
AI BigQuery Gemini May 12, 2025Unleash Row-Level AI in BigQuery: Scalar LLM Functions for Smarter SQL - Bring LLM intelligence to each row: Using BigQuery’s scalar AI functions in everyday SQL tasks.
BigQuery May 12, 2025Observability in Action: A Google Cloud Next demo - An Observability in Action demo at Google Cloud Next 2025 showcased how to interact with metrics and logs using Cloud Monitoring, Cloud Trace, and BigQuery. Attendees used physical buttons to interact with AI models, generating data that was then analyzed using Jupyter Notebooks. The demo highlighted the importance of observability tools for identifying performance bottlenecks and gaining insights from log data.
BigQuery BigQueryML May 12, 2025Supercharge Your Data Quality: Anomaly Detection with BigQueryML - BigQuery ML offers a scalable solution for proactive data quality monitoring using the ML.DETECT_ANOMALIES function. By training machine learning models on historical data, users can identify outliers and anomalies in row counts or metric fluctuations. This approach enables early detection of data quality issues, improving trust and reliability in data assets.
BigQuery Paywall May 12, 2025A Detailed Guide to Key Generation in BigQuery - This blog post will guide you through strategizing surrogate key management in BigQuery.
BigQuery Data Analytics Official Blog May 12, 2025Build, use and share data with data products in BigQuery - Data products in BigQuery is now available in experimental preview. This feature aims to solve the problem of siloed data and inconsistent governance by allowing data producers to bundle and distribute BigQuery tables or views as logical blocks with business context and guarantees. It simplifies the transaction between data producers and consumers, promoting data democratization, reducing redundancy, and accelerating access to insights.
Showing 361–390 of 2427 resources
Useful Links
Contact
Třebanická 183
Prague, Czech Republic
Phone: +420 777 283 075
Email: [email protected]