Welcome to issue #231 March 1st, 2021

News

GKE Autopilot Google Kubernetes Engine Kubernetes Official Blog

Introducing GKE Autopilot: a revolution in managed Kubernetes - GKE Autopilot gives you a fully managed, hardened Kubernetes cluster out of the box, for true hands-free operations.

BigQuery Data Analytics Official Blog

BigQuery delivers a modern view of materialized views - BigQuery materialized views is now GA and provides query cost-optimization and increased performance.

BI Engine BigQuery Data Analytics Official Blog

New in BigQuery BI Engine: faster insights across popular BI tools - BigQuery BI Engine ensures significantly faster queries across popular BI tools.

Business NoSQL Official Blog

MongoDB and Google Extend Partnership To Drive Enterprise Cloud Modernization - MongoDB and Google Partner To Drive Enterprise Cloud Modernization.

Cloud Marketplace Official Blog Terraform

A look at the new Google Cloud Marketplace Private Catalog, now with Terraform support - The latest version of Private Catalog simplifies management for the products you use from Google Cloud Marketplace.

Compute Engine Official Blog

At your service! With schedule-based autoscaling, VMs are at the ready - Schedule-based autoscaling for Compute Engine lets you improve the availability of your workloads by scheduling capacity ahead of anticipated load.

Cloud Monitoring Cloud Operations Monitoring Official Blog

Increasing limits for three key Cloud Monitoring features - Cloud Monitoring now supports more projects and custom metric descriptors, and retains metrics from Compute Engine agents for longer.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

Cloud Operations Official Blog SRE

With SRE, failing to plan is planning to fail - The process of becoming a successful Site Reliability Engineering shop starts well before you take your first class or read your first manual.

Cloud Operations Google Kubernetes Engine Kubernetes Official Blog

Troubleshooting services on Google Kubernetes Engine by example - Learn how to deploy a sample app to your cluster and configure an alerting policy that will notify you if there are any container restarts observed.

Billing Official Blog

Stay informed by customizing your budgets - TL;DR - By default, budget alerts only go to Billing Account Administrators and Billing Account Users. You can easily add up to 5 more custom emails by using a Cloud Monitoring channel. Try to send to groups, not individuals!

Networking Official Blog

What’s in a name? Understanding the Google Cloud network “edge” - The network edge is in the eye of the beholder. Learn about Google Cloud’s various network edge points and how traffic handoffs occur for each one.

Cloud SDK IAM Security

Identify Unused Service Accounts in GCP - Easily find and identify unused service accounts across your entire GCP organization.

Anthos DevOps Kubernetes

Anthos lightweight developer sandbox - A quick way to get started using some features of GCP Anthos.

GKE Autopilot Google Kubernetes Engine Kubernetes

Google Kubernetes Engine Autopilot - Autopilot overview.

Google Kubernetes Engine

Google Kubernetes Engine — Autopilot - Autopilot overview with a demo.

Security

How to build an accountability data lake on Google Cloud Platform in 30 minutes - How to activate Audit Logs for an organization.

Artifact Registry Docker

Using Public and Private Docker Images from GCP Artifact Registry - Using selective access control in Artifact Registry.

App Development, Serverless, Databases, DevOps

Cloud Identity Aware Proxy Cloud Run

Securing Load balanced Authenticated Cloud Run Services using IAP - When designing your cloud run services, you should always consider following points.

Apigee Dialogflow GCP Experience Official Blog

Conversational AI with Apigee API Management for enhancing customer experiences - How Google Cloud’s Dialogflow and Apigee make it easy to deploy interactive virtual agents.

Firebase Official Blog

App monetization made easy: Using Firebase, AdMob, and Google Analytics to uncover your optimal ads strategy

Official Blog Serverless Workflows

3 common serverless patterns to build with Workflows - Learn how to build powerful serverless applications using these common architectural patterns that are now possible with Workflows and Connectors.

IAM Security

Google Cloud SDK with Service Account on Raspberry Pi - Using a Raspberry Pi to interact with your Google Cloud Platform projects without having to expose your user credentials.

API Cloud Functions Serverless

Crypto updates with Cloud Functions (API’s, Serverless, and Twilio) - Serverless notifications about cryptocurrency data.

Serverless Workflows

Parallel executions with Google Workflows - Example of doing parallel tasks in Cloud Workflows.

Big Data, Analytics, ML&AI

Cloud Composer Cloud Data Fusion Data Analytics Official Blog

Framework for building a configuration driven data lake using Data Fusion and Composer - This article provides a framework for building a configuration driven data lake on Google Cloud using Data Fusion and Composer.

Data Analytics Looker Official Blog

Accelerate data science workflows with Looker, plus free training to help get started - Learn how data analysts and scientists can use Looker to help with data governance, join our training to walk through working examples of how Looker can be leveraged to help automate and productionalize data science workflows.

BigQuery Data Analytics Machine Learning Official Blog

How to use a machine learning model from a Google Sheet using BigQuery ML - By combining Sheets with BigQuery ML, you can bring the power of machine learning to data in your spreadsheet.

Cloud Dataproc Official Blog

Migrating Apache Hadoop to Dataproc: A decision tree - Are you using the Apache Hadoop and Spark ecosystem? Are you looking to simplify the management of resources while continuing to use the same tools? If yes, then Dataproc is the tool to check out. In this blog post, we will briefly cover Dataproc and then highlight the four scenarios to migrate Apache Hadoop cluster to Google Cloud.

Data Analytics Official Blog

Run data science at scale with Dataproc and Apache Spark - Dataproc Hub, a feature now generally available for Dataproc users, provides an easier way to scale processing for common data science libraries and notebooks, govern custom open source clusters, and manage costs so that enterprises can maximize their existing skills and software investments.

BigQuery Machine Learning

How To Load Data Into Google BigQuery on Dataproc and AutoML - Loading data into BigQuery using StreamSets.

AI Platform Data Analytics GPU Official Blog

Scale model training in minutes with RAPIDS + Dask and NVIDIA GPUs on AI Platform - Scale model training from 10 GB to 640 GB in minutes with RAPIDS + Dask and NVIDIA GPUs on AI Platform.

BigQuery

Full relational diagram for Ethereum public data on Google BigQuery - Full relational diagram for Ethereum data with SQL examples covering Internal Transactions, Ethereum Trace, Tokens and Smart Contracts.

Big Data BigQuery Data Science

BigQuery Hack: Flexible Queries For Any Number of Columns - How can we use BigQuery to handle tables with many columns? Here’s how using scripting and table metadata.

AI AI Platform Machine Learning Python

Training PyTorch Transformers on GCP AI Platform - How to use highly popular ML framework PyTorch on AI Platform Training.

Apache Beam Cloud Dataflow Monitoring

Monitoring your Dataflow pipelines - This article gives an overview of the different metrics and logs you can use on Google Cloud Platform to monitor your Dataflow jobs.

BigQuery Data Science

From cURL to Automated Workflow - Creating a pipeline to load financial data into BigQuery and do Analytics.

AI Platform Machine Learning

XGBoost deployment on GCP with Google AI Platform - Reducing the gap between a PoC and an integrated solution for binary classification.

Various

AWS GCP Experience Google Cloud Platform

Google Cloud vs AWS Onboarding Comparison

GCP Certification

How to become certified Google Professional Data Engineer - Path to success and study materials for Data Engineer certification.

GCP Certification Security

Google cloud professional security engineer study guide - Topics covered for Security Engineer certification exam.

Slides, Videos, Audio

GCP Podcast - #248 Cloud Spanner Revisited with Dilraj Kaur and Christoph Bussler.

Kubernetes Podcast - #139 GKE Autopilot, with Yochay Kiriaty.

 

Releases

Anthos Config Management - 1.6.2. Hierarchy Controller now includes a preview of Hierarchical Resource Quotas (HRQs). Config Sync now takes advantage of server-side apply when possible, so will be able to handle custom resources larger than 256 kB on Kubernetes clusters with a version of 1.16 or later. The Anthos Config Management Operator Deployment now specifies resources.limits for config-management-operator:manager.

Anthos - Anthos 1.6.2 is now available.

BI Engine - BigQuery BI Engine now interacts with popular BI tools such as Looker, Tableau, and more, by means of an SQL interface.

BigQuery - The INFORMATION_SCHEMA.TABLES view now includes a DDL column. BigQuery materialized views are now generally available (GA).

BigQuery Transfer - The BigQuery Data Transfer Service's 1-hour minimum file age requirement for transfers from Cloud Storage has been eliminated. The BigQuery Data Transfer Service's minimum interval time between recurring transfers from Cloud Storage has been reduced from one hour to 15 minutes.

Billing - Optimal Recommendations for Compute Engine committed use discounts are now Generally Available.

Cloud Composer - New versions of Cloud Composer images: composer-1.14.4-airflow-1.10.14 composer-1.14.4-airflow-1.10.12 (default) composer-1.14.4-airflow-1.10.10. When an environment update operation cannot start, an error message that lists possible causes for the error is generated. Improved the syncing of DAGs and plugins to the Airflow web server. Added FreeTDS system package to Cloud Composer images. Updated apache-beam package version to 2.24.0 in Airflow 1.10.10 so that Dataflow jobs now correctly create partitioned BigQuery tables. Upgraded apache-airflow-backport-providers-google package to version 2021.2.5 in Airflow 1.10.12 and 1.10.14. The google-cloud-pubsublite package is installed by default in Composer images for Airflow 1.10.12 and 1.10.14. Improved the reliability of environment upgrade operations. Fixed the cause of several update-related errors. When an environment deletion operation fails, a correct GKE error is displayed. The Airflow scheduler liveness checker is now compatible with google-cloud-logging==2.2.0. Fixed an error when some MsSQL operators were not working with Azure instances. Fixed an error in GKE cluster builds. GA: Support for Customer Managed Encryption Keys (CMEK) is now generally available.

Compute Engine - Preview: You can now use the gcloud command-line tool to import images from AWS into Google Cloud.

Data Fusion - Cloud Data Fusion Beta instances (versions 6.1.0.2 and lower that were created before November 21, 2019) will be turned down on March 1, 2021.

Dataproc - New sub-minor versions of Dataproc images: 1.3.85-debian10, 1.3.85-ubuntu18, 1.4.56-debian10, 1.4.56-ubuntu18, 1.5.31-centos8, 1.5.31-debian10, 1.5.31-ubuntu18, 2.0.4-debian10, and 2.0.4-ubuntu18. Image 2.0: Upgraded Spark to 3.1.1 RC2 version. Allow stopping clusters that have autoscaling enabled, and allow enabling autoscaling on clusters that are STOPPED, STOPPING, or STARTING. Deactivated mysql and hive-metastore components for clusters created with a Dataproc Metastore service on an image that has the DISABLE_COMPONENT_HIVE_METASTORE and DISABLE_COMPONENT_MYSQL capabilities. Image 1.3 - 1.5: HIVE-18871: hive on Tez execution error due to set hive.aux.jars.path to hdfs://. Dataproc 2.0 image version will become a default Dataproc image version in 3 weeks on March 15, 2021.

Cloud Firestore - A Firestore connector for Workflows is now available in public preview.

GKE on Prem - Anthos clusters on VMware (GKE on-prem) 1.6.2-gke.0 is now available. Fixes: Fixed a kubelet restarting issue that was found when running workloads that rely on kubectl exec/port-forward/attach, such as Jenkins.

IAM - You can now use Policy Simulator to simulate policy changes before you apply them.

Anthos Migrate - On February 23, 2021, we released Migrate for Anthos 1.6.2. 180576558: Fixed an issue where the Linux discovery tool calculated an incorrect score. Fixed an issue where using an Envoy proxy sidecar, not as part of Istio or Anthos Service Mesh, created networking issues with the migrated workload.

Private Catalog - Private Catalog supports Terraform.

Resource Manager - Project migration between organizations is now a self-serve process in public preview.

Cloud SQL MySQL - Cloud SQL now offers faster maintenance, with average connectivity loss lasting 90 seconds or less on average.

Cloud SQL Postgres - Cloud SQL now offers faster maintenance, with average connectivity loss lasting 90 seconds or less on average. The following PostgreSQL minor versions are now available.

Virtual Private Cloud - Hierarchical firewall policies are now available in General Availability. The ability to connect VM interfaces other than nic0 to a Shared VPC is now available in General Availability.

Anthos GKE deployed on-prem - Anthos clusters on VMware (GKE on-prem) 1.6.2-gke.0 is now available. Fixes: Fixed a kubelet restarting issue that was found when running workloads that rely on kubectl exec/port-forward/attach, such as Jenkins.

Service Mesh - 1.8.x. 1.8.3-asm.2 is now available.

Secret Manager - Event notifications is now available in Preview.

Artifact Registry - Support for Python packages in private PyPI repositories is now in alpha.

Memorystore for Memcached - v1. General Availability release of Memorystore for Memcached.

Anthos GKE on AWS - Anthos clusters on AWS 1.6.2-gke.0 is now available. This release fixes an issue where the management service fails to start when provided with a KMS alias. Bug fixes and security improvements.

 

Latest Issues




Contact

Zdenko Hrček
Třebanická 183
Prague, Czech Republic
Phone: +420 777 283 075
Email: [email protected]