Tag: Serverless Spark

Resources about Serverless Spark collected from the weekly GCP Weekly newsletter — a hand-curated collection of Google Cloud Platform news, technical articles, tutorials, videos and release notes. Each entry below links back to the original GCP Weekly issue and retains its publish date and title, giving you a focused, searchable knowledge base for this topic.

Data Analytics GCP Experience Official Blog Serverless Spark Oct. 5, 2026

Accelerating analytics: PayPal’s journey with Managed Service for Apache Spark - Learn how PayPal migrated analytics workloads to Managed Service for Apache Spark on Google Cloud to scale and reduce operational costs.

Agents Data Analytics Python Serverless Spark Aug. 24, 2026

From prompt engineering to agentic workflow: Building intelligent systems on Dataproc Serverless part 1 - This article explains how to transition from traditional manual development to autonomous agentic workflows by automating the Git-to-Google Cloud Storage build pipeline. Using Python, SBT, and Google Cloud tools, the guide demonstrates how to encapsulate complex compilation steps into a reliable, programmatic format that AI agents can execute via natural language.

Data Analytics Official Blog Serverless Spark Streaming Aug. 24, 2026

Serverless Apache Spark on Google Cloud: Architecture Choices & AI Troubleshooting - Learn to choose Apache Spark deployment models, optimize costs, and troubleshoot serverless pipelines on Google Cloud with Gemini Cloud Assist.

Agents Apache Iceberg AWS Data Analytics Serverless Spark Aug. 3, 2026

The borderless Lakehouse: Bring AWS, Databricks and Snowflake data to your AI agents - The borderless Lakehouse provides secure, bi-directional access via BigQuery and Managed Service for Apache Spark to any Iceberg-compatible engine.

Data Analytics Official Blog Serverless Spark Streaming June 15, 2026

Deep dive: How Lightning Engine delivers 4.9x faster Apache Spark performance - How Lightning Engine for Managed Service for Apache Spark, now GA, supercharges job execution compared with standard and managed Spark alternatives.

Cloud Dataproc Data Analytics Official Blog Serverless Spark Streaming June 8, 2026

What’s new in serverless Managed Service for Apache Spark - Learn about the latest features in serverless Managed Service for Apache Spark runtime version 3.0 enabling a wide range of workflows.

Cloud Dataproc Data Analytics Official Blog Serverless Spark Streaming June 8, 2026

What's new for Managed Service for Apache Spark clusters - Running Managed Service for Apache Spark clusters is faster, easier, and smarter with the latest enhancements announced at Google Cloud Next ‘26.

Cloud Dataproc Serverless Spark SRE May 11, 2026

F1 Telemetry and Tuning for your Spark cluster: the BigQuery Log Analytics setup that costs nothing - This article outlines a cost-effective method for gathering detailed telemetry from Spark clusters on Google Cloud. By leveraging a small custom Spark listener and five GCP Log Analytics queries, it extracts crucial job and cluster events for data-driven tuning and resource allocation. This approach provides actionable insights for autoscaled Dataproc clusters significantly cheaper than traditional BigQuery export pipelines.

Serverless Spark May 4, 2026

Lakehouse Demystified — Part 3: Just enough about Spark notebooks powered by Google’s Managed Service for Apache Spark — serverless interactive sessions - This article details Google Cloud's Managed Service for Apache Spark's serverless interactive sessions, which power Spark notebooks for interactive data analysis and engineering. It highlights how this managed service provides an auto-scaling environment for notebooks, eliminating operational toil and accelerating development.

Apache Iceberg Data Analytics Official Blog Serverless Spark April 27, 2026

The future of data lakehouse: Open and interoperable for the agentic era - Transition to an AI-native, cross-cloud Lakehouse built for agentic AI. Deliver 117% ROI with managed Apache Iceberg, always-on context, and high-performance Spark.

AI Antigravity Serverless Spark March 2, 2026

Connect to a serverless Apache Spark notebook in Google Antigravity - This article provides a comprehensive guide on connecting to a serverless Apache Spark runtime within a notebook environment using Google Antigravity, an agentic development platform. It details the process of installing relevant VSCode extensions like Cloud Code and Jupyter, configuring Spark runtime templates, and executing PySpark code directly within the IDE, even leveraging Antigravity's AI features for code generation.

Serverless Spark Nov. 17, 2025

A Google Engineer’s Take on a Common Spark Problem — And How We’re Fixing It - Google Cloud is developing Autotuning for Serverless Apache Spark to automate performance optimization, addressing common issues like executor sizing, shuffle partitioning, and join strategies.

Data Analytics Gemini Official Blog Serverless Spark Oct. 6, 2025

Connect Spark data pipelines to Gemini and other AI models with Dataproc ML library - A new open-source Python library can simplify MLOps for Dataproc by connecting Apache Spark jobs to Gemini and popular ML frameworks.

BigQuery Data Science Official Blog Serverless Spark Sept. 22, 2025

Announcing the new Practical Guide to Data Science on Google Cloud - Learn how to leverage Google Cloud's BigQuery, Serverless Spark, and Vertex AI for modern data science workflows. This guide covers unified tooling, end-to-end solutions, and real-world use cases with code examples.

Python Serverless Spark Sept. 15, 2025

How to add modules and PIP libraries to Dataproc PySpark Serverless jobs - In this post I explain how to add pip libraries and python modules into Dataproc Spark Serverless jobs.

Cloud Dataproc Data Analytics Official Blog Serverless Spark Sept. 8, 2025

Investigate fast with AI: Gemini Cloud Assist for Dataproc & Serverless for Apache Spark - Struggling with failed or slow Spark jobs? Use Gemini Cloud Assist AI in Dataproc to find root causes, get actionable fixes and boost performance.

BigQuery Cloud Dataproc Cloud Run Serverless Spark July 14, 2025

Submitting a PySpark Job to Serverless Dataproc from a Cloud Run Function - The article discusses how to submit PySpark jobs to Serverless Dataproc from a Python-based Cloud Run function for scalable batch processing on Google Cloud.

Data Analytics Official Blog Serverless Spark Streaming June 2, 2025

Introducing Lightning Engine — the next generation of Apache Spark performance - The new Lightning Engine for Apache Spark improves performance for your lakehouse with a variety of optimization techniques.

BigQuery Serverless Spark March 17, 2025

Spark Stored Procedures in BigQuery: A Practical Guide That Works (Part 1) - A step-by-step practical guide to setting up serverless Spark stored procedures in BigQuery that actually works.

Cloud Dataproc Databricks Serverless Spark Nov. 11, 2024

Integrating Open Source Unity Catalog with GCP workloads - Open source Unity Catalog, a data and AI governance solution, can be integrated with GCP workloads like Spark on Dataproc. This blog provides a step-by-step guide to set up Unity Catalog in GCP, including hosting the Unity Catalog server and webserver UI, and interacting with Unity Catalog using Spark workloads.

Serverless Spark April 22, 2024

Google Serverless Spark: An Overview and How to Optimize Performance - This blog post delves into the characteristics and the advantages of Google Serverless Spark.

BigQuery Official Blog Serverless Spark March 25, 2024

Unify analytics with Spark procedures in BigQuery, now generally available

Jupyter Notebook Python Serverless Serverless Spark Feb. 5, 2024

Paperless: Dataproc Serverless Meets Jupyter - Paperless is a Python package designed to streamline the execution of Jupyter Notebooks with a Spark kernel.

Official Blog Serverless Spark Sept. 11, 2023

Tune Spark properties to optimize Dataproc Serverless jobs - Dataproc Serverless uses Spark properties to determine the compute, memory, and disk resources to allocate to your batch workload.

Official Blog Serverless Spark July 31, 2023

Learn how to run Serverless Spark workloads without provisioning and managing clusters - A new interactive tutorial in the Google Cloud console walks you how to start processing your data with Serverless Spark on Google Cloud.

Serverless Spark Workflows July 3, 2023

Event-driven Data Pipeline with Cloud Workflows and Serverless Spark - In some situations, we need to process individual files as soon as they arrive to our platform in order to minimise the time between….

Cloud Dataproc Serverless Spark April 24, 2023

How to Submit Spark Serverless Jobs, Manage Quota and Capture Errors - Today Dataproc Serverless is the modernest way to run your spark jobs in GCP. It lets you get out of the cluster boundaries, giving the….

BigQuery Serverless Spark Vertex AI April 17, 2023

Oracle to BigQuery: Migrate Oracle to BigQuery using Vertex AI notebooks and GCP Dataproc Serverless - Dataproc Templates, in conjunction with VertexAI notebook and Dataproc Serverless, provide a one-stop solution for migrating data directly from Oracle Database to GCP BigQuery.

Data Analytics Official Blog Serverless Spark April 17, 2023

Best practices of orchestrating Notebooks on Serverless Spark - Shift your notebook workloads to Spark and enjoy automated scaling, easier version tracking and automated logging.

BigQuery Python Serverless Serverless Spark Nov. 7, 2022

Creating Serverless Spark Jobs with Google Cloud - Write scalable Spark applications and data pipelines without any manual infrastructure provisioning or tuning.

Showing 1–30 of 41 resources

 

Latest Issues




Contact

Zdenko Hrček
Třebanická 183
Prague, Czech Republic
Phone: +420 777 283 075
Email: [email protected]