Tag: Cloud Dataproc

Cloud Dataproc Data Science April 1, 2024

Spark Performance Tuning for BigQuery APIs - Dealing with challenges when using Spark for NLP processing.

Cloud Dataproc GCP Experience Official Blog Feb. 26, 2024

Serverless data architecture for trade surveillance at Deutsche Bank - Deutsche Bank uses Google Cloud's BigQuery and Dataproc to streamline trade surveillance. This serverless architecture simplifies data sharing, reduces costs, and allows them to focus on detecting suspicious activity and ensuring regulatory compliance.

Cloud Dataproc Jan. 22, 2024

Infrastructure failures during big data processing - This blog post explains how to handle hardware failures when running Spark jobs.

Cloud Dataproc Official Blog Dec. 11, 2023

Autoscaling Dataproc for Trino workloads - Autoscaler for Trino on Dataproc solution provides reliable autoscaling for Trino on Dataproc without compromising workload execution.

Cloud Dataproc Dec. 11, 2023

Using Spark on Dataproc & Apache Iceberg To Build an Open Lakehouse - Using Spark on Dataproc in GCP for reading and writing from a Lakehouse.

Cloud Dataproc Dec. 3, 2023

A guide to RAID multiple Local SSDs & mount it to Dataproc - A guide to RAID multiple Local SSDs & mount it to Dataproc.

Cloud Dataproc Official Blog Nov. 6, 2023

Reduce costs and improve job run times with Dataproc autoscaling enhancements - This blog post highlights the cost and job performance improvements resulting from Dataproc autoscaling enhancements, without the administrator needing to change the cluster configuration or autoscaling policy.

BigQuery Cloud Dataproc Data Science dbt Python Oct. 16, 2023

Choosing the right tool while building your Data Platform: DBT vs. Spark (By example) - Table of contents.

Cloud Dataproc Aug. 21, 2023

Understanding Driver Pools in Dataproc - Learn about driver pools in Dataproc — a mechanism to scale application concurrency in Dataproc clusters.

Cloud Dataproc Scala Aug. 14, 2023

Spark Scala job with Dataproc Serverless - 1. Explanation of the use case presented in this article.

Cloud Dataproc Security June 26, 2023

Access Control on Dataproc for Hive and Spark jobs - What are the basics of access control? What options do we have on Dataproc for properly handling access control?

Cloud Dataproc Data Analytics Official Blog June 19, 2023

Statsig unlocks new features by migrating Spark to BigQuery - Migrating to BigQuery from Spark helped Statsig to develop new features for customers and help them run scalable experimentation programs.

Cloud Bigtable Cloud Dataproc Cloud Pub/Sub May 1, 2023

Stream data from Pub/Sub Lite to Bigtable using Dataproc Serverless - This blog post explains how to stream data from a Pub/Sub Lite to BigTable using Dataproc.

Cloud Dataproc Serverless Spark April 24, 2023

How to Submit Spark Serverless Jobs, Manage Quota and Capture Errors - Today Dataproc Serverless is the modernest way to run your spark jobs in GCP. It lets you get out of the cluster boundaries, giving the….

Cloud Dataproc April 10, 2023

Understanding CPU Oversubscription in Dataproc/Hadoop - This post explains the what, how and the why about CPU oversubscription in Hadoop clusters. It attempts to clear general misconceptions.

Airflow Big Data Cloud Dataproc Cloud Storage March 13, 2023

Event Driven Data Processing on Google Cloud Platform - An example of event-driven data pipeline.

Cloud Dataproc Terraform March 13, 2023

Creation of Google Cloud Platform Dataproc Workflow templates via Terraform

BigQuery Cloud Dataproc Tutorial March 6, 2023

Creating Parameterized Spark Jobs on Ephemeral DataProc Clusters - This guide aims to show you how to easily parameterize a Spark job on GCP DataProc and then run it on an ephemeral cluster.

Cloud Dataproc Data Analytics Official Blog Jan. 16, 2023

Run faster and more cost-effective Dataproc jobs - Are your Dataproc jobs running too slow? Do you need to optimize the costs of your Dataproc job strategy? This blog explores a step-by-step approach to improving your Dataproc jobs.

BigQuery Cloud Dataproc Jan. 9, 2023

Import Data From Postgres to BigQuery in Parallel via Dataproc - Leveraging the Power of PySpark to fetch data from Postgres in Parallel without Primary key.

Cloud Dataproc Data Analytics Official Blog Dec. 12, 2022

Best practices of Dataproc Persistent History Server - The challenge with ephemeral clusters and serverless Spark is that you will lose the application logs when the cluster machines are deleted after the job. Persistent History Server (PHS) enables you to monitor Spark applications running on different ephemeral clusters or serverless Spark.

Cloud Composer Cloud Dataproc Serverless Nov. 14, 2022

Use Composer for Dataproc Serverless workloads - Using Composer to run Dataproc jobs.

Cloud Dataproc Data Analytics Official Blog Oct. 31, 2022

Best practices for migrating Hadoop to Dataproc by LiveRamp - Liveramp sharing best practices for migrating apache hadoop from on-prem to Google Cloud Dataproc.

Apache Beam Cloud Dataproc Data Analytics Jupyter Notebook Official Blog Oct. 24, 2022

Run interactive pipelines at scale using Beam Notebooks - Run Apache Beam pipelines for ML inference interactively in Jupyter Notebooks with FlinkRunner at scale using Dataproc on Google Cloud under the hood.

BigQuery Cloud Dataproc Jupyter Notebook Oct. 17, 2022

Delta tables with Dataproc, Jupyter (and BigQuery) - Loading data from Delta tables to BigQuery in Dataproc.

Cloud Dataproc GCP Experience Python Sept. 12, 2022

Why we don’t use Spark - A use case of migrating from Dataproc to Kubernetes.

BigQuery Cloud Dataproc Serverless Spark July 18, 2022

Processing data from Hive to BigQuery using PySpark and Dataproc Serverless - How to run a batch workload to process data from an Apache Hive table to a BigQuery table, using PySpark and Dataproc Serverless.

Big Data Cloud Dataproc June 20, 2022

Big Data Processing using Google Dataproc - Google Dataproc is a very powerful option for Hadoop and Spark applications-enabled clusters.

Big Data Cloud Dataproc June 6, 2022

Tuning Spark Applications to Efficiently Utilize Dataproc Cluster - Have you recently migrated your Spark application from the on-prem Yarn cluster to Dataproc? Then this blog post might help you to tune your Spark applications to efficiently utilize the GCP Dataproc and save cost.

Cloud Dataproc Python May 16, 2022

Churn Prediction with PySpark and Google Cloud Dataproc - Using PySpark on Cloud Dataproc to predict users' churn.

Cloud Dataproc Kubernetes Official Blog Serverless Spark April 18, 2022

Running Spark on Kubernetes with Dataproc - Derive benefits from fully automated, most scalable and cost optimized Kubernetes service for your Spark and open source workloads.

Cloud Dataproc Serverless Spark April 18, 2022

Processing databricks Delta Lake data in Google Cloud Dataproc Serverless for Spark - Migrating from Dataproc to Serverless Spark.

BigQuery Cloud Dataproc Official Blog Vertex AI April 11, 2022

Announcing Serverless Spark components for Vertex AI Pipelines - You can use Vertex AI Pipelines to automate ML workflows in conjunction with Dataproc for running serverless Spark workloads!

Cloud Dataproc April 4, 2022

Dataproc and Apache Spark tuning - When you migrate Spark jobs from on-premise Hadoop cluster to the Cloud Dataproc ephemeral clusters you should not lift and shift spark.properties. It is much easier to use Spark dynamic allocation to fill the allocated Dataproc cluster capacity.

Cloud Dataproc Cloud Spanner Serverless Spark April 4, 2022

Cloud Spanner export query results using Dataproc Serverless - Exporting data for a Spanner Table or SQL Query using Dataproc Serverless.

Cloud Dataproc Python March 28, 2022

Running pyspark jobs on Google Cloud using Serverless Dataproc - Run Spark batch workloads without having to bother with the provisioning and management of clusters!.

Cloud Dataproc Data Analytics Official Blog Oct. 25, 2021

Spark on Google Cloud: Serverless Spark jobs made seamless for all data users - Spark on Google Cloud allows data users of all levels to write and run Spark jobs that autoscale, from the interface of their choice, in 2 clicks.

Big Data BigQuery Cloud Dataproc GCP Experience Sept. 27, 2021

Comparing BigQuery Processing and Spark Dataproc - Paypal's approaches for evaluation for migrating processes from on-prem to GCP.

Cloud Dataproc Data Analytics GPU HPC Official Blog Aug. 30, 2021

Single-cell genomic analysis accelerated by NVIDIA on Google Cloud - Learn about single-cell genomic analysis on Google Cloud using NVIDIA and Dataproc.

BigQuery Cloud Dataproc GCP Experience Aug. 9, 2021

Comparing BigQuery Processing and Spark Dataproc - Different approaches that were evaluated for migrating Paypal's processes from on-prem to GCP.

Cloud Dataproc Data Science Aug. 2, 2021

Creating a Dataproc cluster: considerations, gotchas & resources - This article discusses focus areas users should consider in their efforts to successfully create a reliable, reproducible, and consistent cluster.

Cloud Dataproc Official Blog July 12, 2021

How to build an open cloud datalake with Delta Lake, Presto & Dataproc Metastore - Building an Open Data Lake with Apache Spark for data processing, Presto as a query engine and Open Formats such as Delta Lake for storing all data.

Cloud Composer Cloud Dataflow Cloud Dataproc Official Blog June 22, 2021

Orchestrating your data workloads in Google Cloud - The Data Orchestration is becoming more important as workflows expand and become more complex on the Cloud. This blog touches on how to tackle data orchestration in GCP using Cloud Composer!

Cloud Dataproc Data Analytics Official Blog June 22, 2021

Dataproc best practices guide - Best practices for Storage, Compute and Operations when using Dataproc for running Hadoop- or Spark-based workloads.

Airflow Cloud Dataproc Data Science June 14, 2021

Apache Airflow + GCP Dataproc via DataProcSparkOperator - Doing integration with Cloud Dataproc and exploring DataProcSparkOperator running Airflow.

Cloud Dataproc Tutorial May 31, 2021

Persistent Spark History Server with Transient Dataproc clusters - The article explains how to set up a Persistent Spark History Server which can collect event logs from multiple Spark applications running on multiple transient clusters and can show the Spark UI when the application finishes.

Cloud Dataproc Data Analytics Official Blog April 19, 2021

Broadcom improves customer threat protection with flexible data management - Broadcom uses decentralized data lake infrastructure to speed development of security analytics.

Big Data Cloud Dataproc Python April 12, 2021

How to migrate your on-premise pyspark jobs to GCP using Dataproc Workflow Templates using Dataproc Workflow Templates with Production-Grade Best Practices Standards - Complete pattern example of how to migrate (or create from scratch) pyspark jobs to GCP with Dataproc Workflow Templates.

Cloud Dataproc Data Analytics Official Blog April 5, 2021

Data Lake management just got easier with Dataproc Metastore GA - Today, we are excited to announce the general availability of Dataproc Metastore. A fully managed, serverless technical metadata repository based on the Apache Hive metastore.

Cloud Dataproc March 29, 2021

Machine Learning with Spark on Google Cloud Dataproc Workshop - In this workshop, you will learn how to prepare the Spark interactive shell on a Google Cloud Dataproc cluster, create a training dataset for machine learning using Spark, develop a logistic regression machine learning model using Spark, and evaluate the predictive behaviour of a machine learning model using Spark on Google Cloud Datalab.

Cloud Dataproc March 15, 2021

Running ETL Spark Job through Dataproc (an ephermal cluster)with Workflow Templates - An example of Spark job running on Cloud Dataproc.

Cloud Dataproc Security March 8, 2021

Securing Presto on GCP DataProc with username and password over HTTPS - A walk through the steps of securing a Presto cluster deployed on GCP DataProc with a username and password authentication over HTTPS.

Cloud Dataproc Official Blog March 1, 2021

Migrating Apache Hadoop to Dataproc: A decision tree - Are you using the Apache Hadoop and Spark ecosystem? Are you looking to simplify the management of resources while continuing to use the same tools? If yes, then Dataproc is the tool to check out. In this blog post, we will briefly cover Dataproc and then highlight the four scenarios to migrate Apache Hadoop cluster to Google Cloud.

Cloud Dataproc Feb. 22, 2021

Active Directory Setup with Kerberized Dataproc Cluster - A manual process of setting authentication from Active Directory to Dataproc.

Cloud Dataproc Cloud SQL Cloud Storage Machine Learning Jan. 4, 2021

Building a Product Recommendation Engine on Google Cloud’s Platform - Implementing recommendation engine using Cloud DataProc, Cloud SQL and Cloud Storage.

Cloud Dataproc Data Science Machine Learning Jan. 4, 2021

All you need to know about Google Cloud Dataproc? - Managed Hadoop & Spark #GCPSketchnote.

Cloud Dataproc Data Analytics Official Blog Dec. 21, 2020

Dataproc Metastore: Fully managed Hive metastore now in public preview - Dataproc Metastore lets you use your Apache Hive metastore to simplify technical metadata management when you’re building a data lake on Google Cloud.

Cloud Dataproc Dataproc Hub Machine Learning Official Blog Dec. 14, 2020

Dataproc Hub makes notebooks easier to use for machine learning - Dataproc Hub, now generally available, makes it easy to use open source, notebook-based machine learning on Google Cloud, powered by Spark.

Cloud Dataproc Data Analytics Official Blog Python Dec. 14, 2020

Improve the data science experience using scalable Python data processing - Announcement of the Dask support for Dataproc, Google Cloud’s fully managed Apache Hadoop and Apache Spark service, via a new initialization action.

Big Data Cloud Dataproc Data Analytics Official Blog Dec. 7, 2020

Best practices to use Apache Ranger on Dataproc - Run managed open source like Apache Hadoop and Spark in the cloud. Get tips on secure deployment with Dataproc and the Apache Ranger authorization OSS.

Cloud Dataproc Data Analytics Official Blog Nov. 16, 2020

Dataproc cooperative multi-tenancy - How you can use Dataproc Cooperative Multi-Tenancy to share Dataproc clusters across multiple users.

BigQuery Cloud Dataflow Cloud Dataproc Python Nov. 9, 2020

BigFlow — a Python framework for data processing on GCP - BigFlow is a Python framework for big data processing on GCP.

Big Data Cloud Dataproc Data Analytics Official Blog Oct. 26, 2020

Preparing for serverless big data open source software - Serverless capabilities at Google Cloud continue to develop, and serverless is now meeting open source as tools like Dataproc let you build on your open foundation in the cloud.

Cloud Dataproc Data Analytics Official Blog Oct. 19, 2020

New Dataproc optional components support Apache Flink and Docker - Run native Apache Spark and Hadoop clusters on Dataproc fast and cost-effectively. New optional components for Docker and Flink available.

BigQuery Cloud Dataproc Data Studio Oct. 19, 2020

Explore & Visualize 200+ Years of Global Temperature Using Apache Spark, BigQuery, and Google Data Studio - Visualize observable changes in global temperature using NOAA’s historical weather data.

Cloud Dataproc Sept. 28, 2020

How to setup auto scalable Google Dataproc cluster? - Creating a Dataproc cluster with auto scaling policy.

Cloud Dataproc Sept. 28, 2020

How Google Dataproc cluster auto scales? - An example of an autoscaling Dataproc cluster.

Cloud Dataproc Sept. 14, 2020

Long-Running Spark Jobs on GCP using Dataproc with Preemptible Instances - Validating Spark Jobs on Dataproc Endure Preemptible Instance Recycling.

BigQuery Cloud Dataproc Data Studio Jupyter Notebook Python Aug. 31, 2020

How to Set up a COVID-19 Workflow and Dashboard Using the Google Cloud Platform - Building a data pipeline to process and visualize data regarding COVID-19.

Cloud Dataproc Data Science Jupyter Notebook Tutorial July 27, 2020

Getting Started with Jupyter + Spark on the Cloud in 2020 - Spinning up Spark clusters with Jupyter on Cloud Dataproc.

Cloud Dataproc Data Analytics Official Blog July 3, 2020

Presto optional component now available on Dataproc - The Presto query engine optional component is now available for Dataproc, Google Cloud’s fully managed Spark and Hadoop cluster service.

Cloud Dataproc Data Analytics Official Blog July 3, 2020

Dataproc Metastore: Fully managed Hive metastore now available for alpha testing - Dataproc Metastore is a fully managed open source Apache Hive metastore service, so you can easily build data lakes on Google Cloud.

Big Data Cloud Dataproc June 22, 2020

Sqoop Data Ingestion on GCP - Using Apache Sqoop (bulk data transfer) in Cloud Dataproc.

Cloud Dataproc Data Analytics Official Blog June 22, 2020

Introducing Spark 3 and Hadoop 3 on Dataproc image version 2.0 - Dataproc image version 2.0 is ready for testing, with Spark 3 and Hadoop 3 capabilities for open source data and analytics testing.

BigQuery Cloud Dataproc June 22, 2020

Graph data analysis with Cypher and Spark SQL on Cloud Dataproc - How to read in BigQuery data and use Spark SQL and the Morpheus library to carry out graph data analysis.

Cloud Dataproc Data Analytics Jupyter Notebook Official Blog June 1, 2020

Combining the power of Apache Spark and AI Platform Notebooks with Dataproc Hub - Dataproc Hub: Administering Jupyter notebooks for Spark workloads on Dataproc.

Cloud Dataproc Data Analytics Official Blog June 1, 2020

Migrating Apache Hadoop clusters to Google Cloud - Moving on-premises Hadoop clusters can help your data analysts, data scientists, and more to work faster and more quickly in the cloud.

Cloud Dataproc Data Analytics Official Blog May 25, 2020

Burst data lake processing to Dataproc using on-prem Hadoop data - Use Dataproc and Alluxio to burst workload processing to cloud from Hadoop on-prem data stores.

Big Data BigQuery Cloud Dataproc Jupyter Notebook May 25, 2020

Apache Spark BigQuery Connector — Optimization tips & example Jupyter Notebooks - Learn how to use the BigQuery Storage API with Apache Spark on Cloud Dataproc.

Big Data BigQuery Cloud Dataproc May 18, 2020

Import SQL Server data in BigQuery - A list of four approaches for a one-off data dumps from a RDBMS like SQL Server to BigQuery.

Cloud Dataproc Machine Learning May 18, 2020

MLOps series #1 : Batch scoring with Mlflow Model (Mleap flavor) on Google Cloud Platform - Deploying ML model from Databrics cluster to Cloud Dataproc.

Big Data Cloud Dataproc May 4, 2020

Migrating Data Processing Hadoop Workloads to GCP - Intro to Dataproc as well as tips for best usage.

Cloud Dataproc Visualization May 4, 2020

Connecting your Visualization Software to Hadoop on Google Cloud - The article explains how to set up architecture for visualization with Hadoop ecosystem on GCP.

Cloud Dataproc Visualization May 4, 2020

Connecting your Visualization Software to Hadoop on Google Cloud - In part 2, steps to set up an environment that will hold data for visualization are explained.

Billing Cloud Dataproc Data Analytics Official Blog May 4, 2020

Optimize Dataproc costs using VM machine type - Try optimizing big data clusters from Spark, Hadoop, and Presto with fully managed Dataproc. Choosing VMs wisely can save on Dataproc costs.

BigQuery Cloud Dataproc Python April 27, 2020

Apache Spark & Google Cloud DataProc - The article goes through a process of setting Dataproc cluster and executing batch Spark job which stores results in BigQuery.

Cloud Dataproc Java April 13, 2020

How to run a Java 11 Spark Job on Google Cloud Dataproc - This tutorial shows how to set up Google Cloud Dataproc Spark jobs to run software compiled in Java 11.

Cloud Dataproc Data Analytics GPU Official Blog April 13, 2020

Machine learning with XGBoost gets faster with Dataproc on GPUs - Machine learning workloads can move a lot faster when run on GPUs vs. CPUs. See how to do it with NVIDIA, XGBoost and Dataproc for ML model building.

BigQuery Cloud Dataproc Data Science Jupyter Notebook March 16, 2020

Apache Spark and Jupyter Notebooks made easy with Dataproc component gateway - Make use of the new Dataproc optional components and component gateway features to easily use Jupyter Notebooks.

AWS Cloud Dataproc GCP Experience NoSQL Python March 9, 2020

Cross-Cloud HBase/Phoenix Data Migration - Using Cloud Dataproc to run Spark job which migrates data from AWS to GCP.

Beginner Cloud Composer Cloud Dataproc Data Science March 9, 2020

A gentle introduction to Data Workflows with Apache Airflow and Apache Spark - A tutorial on using Cloud Composer (Airflow) to launch Spark jobs on Cloud Dataproc.

Cloud Dataproc Machine Learning Feb. 24, 2020

What I learned about deploying Machine Learning application - A tutorial on building custom ML training workflow using Google Cloud Platform.

Cloud Dataproc Feb. 24, 2020

Apache Druid Production Setup in Google Cloud Platform with Dataproc cluster — Part 1 - Setting up Apache Druid (a real-time analytics database designed for fast slice-and-dice analytics (“OLAP” queries) on large data sets).

Cloud Dataproc Data Analytics Official Blog Jan. 6, 2020

Getting started with new table formats on Dataproc - You can now use table format projects Apache Iceberg and Delta Lake with Google Cloud’s Dataproc, built to run Hadoop systems in the cloud.

Cloud Dataproc Dec. 9, 2019

Efficiently managing Dataproc cluster - Creating and stoping Dataproc cluster triggered by a new file in Cloud Storage.

Big Data BigQuery Cloud Dataproc Nov. 25, 2019

Querying External Data with BigQuery - Demonstration of BigQuery querying Parquet files from Google Cloud Storage.

BigQuery Cloud Composer Cloud Dataproc Oct. 6, 2019

Running Spark on Dataproc and loading to BigQuery using Apache Airflow - Example of Airflow data pipeline setup for Dataproc and BigQuery.

Cloud Dataproc Data Analytics Google Kubernetes Engine Official Blog Sept. 16, 2019

Cloud Dataproc Spark Jobs on GKE: How to get started - Cloud Dataproc offers alpha access to Spark jobs on Google Kubernetes Engine (GKE) for data analytics with speed and scale.

Cloud Dataproc Data Analytics Google Kubernetes Engine Official Blog Sept. 16, 2019

Modernize Apache Spark with Cloud Dataproc on Kubernetes - Alpha availability of Cloud Dataproc for Kubernetes.

Cloud Dataproc July 8, 2019

Introducing advanced security options for Cloud Dataproc, now generally available - With Kerberos and Hadoop secure mode, you can migrate your existing Hadoop security controls directly into the cloud without having to make changes to your security policies and procedures.

Cloud Dataproc Data Analytics Official Blog June 17, 2019

7 best practices for running Cloud Dataproc in production - Best practices to develop reliant and stable production processes with Cloud Dataproc.

Cloud Dataproc Data Science June 17, 2019

Scale out RAPIDS on Google Cloud Dataproc - Scaling GPU data jobs on Cloud Dataproc.

Cloud Dataflow Cloud Dataproc April 29, 2019

Hadoop Ecosystem in Google Cloud Platform - Overview of Hadoop-like products on Google Cloud Platform.

Cloud Dataproc Official Blog April 22, 2019

New open-source tools in Cloud Dataproc process data at cloud scale - Overview of the most recent Cloud Dataproc features announced on Next '19.

Cloud Dataproc April 8, 2019

Persisting Application History from Ephemeral Clusters on Google Cloud Dataproc - This post outlines a pattern to have a single-node long-running Dataproc cluster that acts as an Application History Server for a shorter lived cluster.

BigQuery Business Cloud Dataproc April 8, 2019

The Economics of Modernizing Your Enterprise Data Warehouse - Results of studies quantifying the economic value of Google data analytics services.

Cloud Dataproc Official Blog Feb. 4, 2019

10 tips for building long-running clusters using Cloud Dataproc - Tips and recommendations for using Cloud Dataproc in a non-ephemeral model.

Cloud Dataproc Official Blog TensorFlow Feb. 4, 2019

AI in Depth: Cloud Dataproc meets TensorFlow on YARN: Let TonY help you train right in your cluster - How to install a Hadoop cluster for LinkedIn open-source project TonY (TensorFlow on YARN).

Beginner Cloud Dataproc Tutorial Dec. 31, 2018

PySpark Sentiment Analysis on Google Dataproc - A Step-by-Step Tutorial on PySpark Sentiment Analysis on Google Dataproc.

Cloud Dataproc Official Blog Dec. 24, 2018

Announcing the beta release of SparkR job types in Cloud Dataproc - Beta release of SparkR jobs on Cloud Dataproc, which is the latest chapter in building R support on GCP.

Cloud Dataproc Dec. 24, 2018

Using the Google Cloud Dataproc WorkflowTemplates API to Automate Spark and Hadoop Workloads on GCP - Examine the Cloud Dataproc WorkflowTemplates API to more efficiently and effectively automate Spark and Hadoop workloads.

Cloud Dataproc Java Python Dec. 17, 2018

Big Data Analytics with Java and Python, using Cloud Dataproc, Google’s Fully-Managed Spark and Hadoop Service - Exploring Cloud Dataproc’s ability to quickly and efficiently run Spark jobs written in Java and Python.

Cloud Dataproc Nov. 26, 2018

Starting to develop in PySpark with Jupyter installed in a Big Data Cluster - Steps to start using Jupyter notebooks for PySpark in a Data Proc Cluster in GCP.

Cloud Dataproc Official Blog Nov. 19, 2018

New report examines the economic value of Cloud Dataproc’s managed Spark and Hadoop solution - ESG recently published a blog and an Economic Value Validation (EVV) report commissioned by Google, which examines the value delivered by Cloud Dataproc.

Cloud Dataproc Stackdriver Nov. 19, 2018

Get more value out of your application logs in Stackdriver - How to get your individual application logs into Stackdriver Logging and tagged separately into their own logger using Google Dataproc 1.2 and google-fluentd.

Cloud Dataproc Official Blog Nov. 19, 2018

Help for slow Hadoop/Spark jobs on Google Cloud: 10 questions to ask about your Hadoop and Spark cluster performance - How to improve your Hadoop and Spark job performance on Google Cloud Platform.

Advanced Cloud Dataproc Official Blog Tutorial Sept. 17, 2018

A flexible way to deploy Apache Hive on Cloud Dataproc - Tutorial shows how to use Apache Hive on Cloud Dataproc in an efficient and flexible way by storing Hive data in Cloud Storage and hosting the Hive metastore in a MySQL database on Cloud SQL and with that providing certain advantages.

Cloud Dataproc Tutorial Sept. 10, 2018

Run your Spark and Hadoop jobs as a Service with Dataproc Workflow Templates - Demo to create a workflow template, add one or more jobs to the template.

Cloud Dataproc Sept. 1, 2018

Convert CSV to Parquet using Hive on Cloud Dataproc - How to convert CSV to Parquet using Hive on Cloud Dataproc.

Cloud Dataproc Java Official Blog Aug. 20, 2018

Managing Java dependencies for Apache Spark applications on Cloud Dataproc - Include Java dependencies for Apache Spark applications on Cloud Dataproc.

Cloud Dataproc Official Blog July 16, 2018

Using instance metadata in Cloud Dataproc initialization actions - How to use instance metadata in Cloud Dataproc initialization actions.

Cloud Dataproc Cloud Pub/Sub Official Blog July 9, 2018

Using Apache Spark DStreams with Cloud Dataproc and Cloud Pub/Sub - Using Cloud Dataproc for running a Spark streaming job that processes messages from Cloud Pub/Sub in near real-time.

Cloud Dataproc Cloud Pub/Sub Tutorial July 2, 2018

Using Apache Spark DStreams with Cloud Dataproc and Cloud Pub/Sub - This tutorial shows how to deploy an Apache Spark DStreams app on Cloud Dataproc and process messages from Cloud Pub/Sub in near real time.

Cloud Dataproc Machine Learning Official Blog April 9, 2018

Using BigDL for deep learning with Apache Spark and Google Cloud Dataproc - BigDL, a distributed deep learning library can be used to write deep learning applications as standard Spark programs in either Scala or Python and directly run them on top of Cloud Dataproc clusters.

Cloud Dataproc Google Kubernetes Engine Official Blog April 2, 2018

Testing future Apache Spark releases and changes on Google Kubernetes Engine and Cloud Dataproc - Know how to test future Apache Spark releases and changes on Google Kubernetes Engine and Cloud Dataproc.

Cloud Dataproc March 19, 2018

Migrating On-Premises Hadoop Infrastructure to Google Cloud Platform - Solution article about migrating on-premises Hadoop infrastructure to Google Cloud Platform.

Cloud Dataproc Feb. 12, 2018

Autoscaling Google Dataproc Clusters - Create and run Apache Spark and Apache Hadoop clusters in a simple and very cost-efficient way using Cloud Dataproc.

Cloud Dataproc Jan. 29, 2018

Updating Cloud Dataproc for faster speeds and more resiliency - Take a look at how Cloud Dataproc now supports high availability (HA) and offer an option for greater performance.

Cloud Dataproc Jan. 29, 2018

Google Cloud Platform POC Part 1 — hadoop distcp to Google cloud storage - Using Cloud Dataproc: challenges faced and solutions.

Cloud Dataproc Jan. 29, 2018

Google Cloud Platform POC Part 2 — Create hive schema, run a spark job, scale the cluster - Using Hive and Spark on Dataproc cluster.

App Engine BigQuery Cloud Dataflow Cloud Dataproc GCP Experience Dec. 18, 2017

How We Implemented a Fully Serverless Recommender System Using GCP - In depth description with code samples of implementing recommendation (serverless) system on Google Cloud Platform.

Cloud Dataproc Tutorial Nov. 27, 2017

Launch a Hadoop Cluster in 90 Seconds or Less in Google Cloud Dataproc! - Step by step tutorial about setting Dataproc (Hadoop cluster).

Cloud Dataproc Oct. 30, 2017

The Data Engineering team at Cabify - Article describes first thoughts of using Google Cloud Dataproc and BigQuery.

Cloud Dataproc Oct. 16, 2017

Control and granularity with Spark and Hadoop on Cloud Dataproc - 3 improvement in Cloud Dataproc: granular IAM, scheduled deletion, per-second billing

Big Data Cloud Dataproc Aug. 20, 2017

Easier integration with Apache Spark and Hadoop via Google Cloud Dataproc Job IDs and Labels - Best practices to use Job IDs and labels

Cloud Dataproc July 31, 2017

Cloud Dataproc is now even faster and easier to use for running Apache Spark and Apache Hadoop - Updates and improvements for Google Cloud Dataproc

Cloud Dataproc June 12, 2017

Fastest track to Apache Hadoop and Spark success: using job-scoped clusters on cloud-native architecture

Cloud Dataproc May 1, 2017

How Feature Engineering can help you do well in a Kaggle competition - Part I - Using Google Cloud Platform for Kaggle challenge

Cloud Dataflow Cloud Dataproc Cloud Datastore March 27, 2017

Example to Integrate Spark Streaming with Google Cloud at Scale - Github repository which contains example to integrate Spark Streaming with Google Cloud products. The streaming application pulls messages from Google Pub/Sub directly without Kafka, using custom receivers. When the streaming application is running, it can get entities from Google Datastore and put ones to Datastore.

Cloud Dataproc March 6, 2017

Google Cloud Platform for data scientists: using Jupyter Notebooks with Apache Spark on Google Cloud - Analyzing data (NYC Taxi trips) on Google Cloud Dataproc with Spark and Jupiter

Cloud Dataproc Official Blog

Customer Managed Encryption Keys (CMEK) for Dataproc is now generally available - The latest is Cloud Dataproc Customer Managed Encryption Keys (CMEK), a feature that is now generally available.

Cloud Dataproc Official Blog

Extending the SQL capabilities of your Cloud Dataproc cluster with the Presto optional component - Presto Distributed SQL Query Engine for Big Data is now available in public beta as an optional component for Cloud Dataproc.

Cloud Dataproc

Massively Parallel Computations using DataProc - Calculating integral with Monte Carlo using Spark on Dataproc as an example of parallel computation.