Cloudera Tutorials

Optimize your time with detailed tutorials that clearly explain the best way to deploy, use, and manage Cloudera products. Login or register below to access all Cloudera tutorials.

First Name

Last Name

Job Title

Business Email

Company

Phone

Country

Yes, I would like to be contacted by Cloudera for newsletters, promotions, events and marketing activities. Please read our privacy and data policy.

Yes, I consent to my information being shared with Cloudera's solution partners to offer related products and services. Please read our privacy and data policy.

I agree to Cloudera's terms and conditions.

Back to main tutorial page

Cloudera acquires Octopai's platform to enhance metadata management capabilities

Read the press release

Introduction

Experience the benefits of having access to a hybrid cloud solution. Using Cloudera AI (formerly Cloudera Machine Learning) on Cloudera, see how an AI workload compares running on-premises versus leveraging computational resources in the cloud.

Prerequisites

Have access to Cloudera on cloud
Have created a Cloudera workload User
Ensure proper Machine Learning role access
- MLUser: ability to run workloads
- MLAdmin: ability to create and delete workspaces

Outline

Watch Video
Download Assets
Setup Cloudera AI
Create Project
Run Experiments
Summary
Further Reading

Watch Video

The video below provides a brief overview of what is covered in this tutorial:

Download Assets

There are two (2) options in getting assets for this tutorial:

Download a ZIP file

It contains only necessary files used in this tutorial. Unzip tutorial-files.zip and remember its location.

Clone our GitHub repository

It provides assets used in this and other tutorials; organized by tutorial title.

Set up Cloudera AI

Provision Machine Learning workspace

If your environment doesn’t already have a Machine Learning workspace provisioned, let’s provision it.

Select Machine Learning from the Cloudera home page:

In the ML Workspaces section, select Provision Workspace.

Two simple pieces of information are needed to provision an ML workspace - the Workspace Name and the Environment name. For example:

Workspace Name: cml-tutorial
Environment: <your environment name>
Select Provision Workspace

Add Resource Profile for additional vCPU / Memory

Beginning from the ML Workspaces section, open your workspace by selecting its name, cml-tutorial.

In the Site Administration section, select Runtime/Engine. Create a new resource profile using:

vCPU: 2

Memory (GiB): 16

Select Add

Create Project

Beginning from the ML Workspaces section, open your workspace by selecting its name, cml-tutorial.

Select New Project.

Complete the New Project form using:

Project Name: Transfer Learning
Project Description:
A project showcasing the speed improvements of running heavy AI workloads on-premises versus using GPU resources on the cloud.
Initial Setup: Local Files
Upload or Drag-Drop cml-files folder you downloaded earlier

Select Create Project

Run Experiments

We will create three (3) experiments to verify speed improvements of AI workload and see the effect GPUs have on training the model.

Beginning from the Projects section, select the project name, Transfer Learning.

In the Experiments section, select Run Experiment and complete the form as follows:

Script: main.py
Kernel: Python 3.8
Edition: Nvidia GPU
Version: 2021.06
Resource Profile: 2 vCPU / 16 GiB Memory, 0 GPUs
Comment: 0 GPU
Select Start Run

Similarly, let’s create an experiment using 1 GPUs:

Script: main.py
Kernel: Python 3.8
Edition: Nvidia GPU
Version: 2021.06
Resource Profile: 2 vCPU / 16 GiB Memory, 1 GPUs
Comment: 1 GPU
Select Start Run

Similarly, let’s create an experiment using 2 GPUs:

Script: main.py
Kernel: Python 3.8
Edition: Nvidia GPU
Version: 2021.06
Resource Profile: 2 vCPU / 16 GiB Memory, 2 GPUs
Comment: 2 GPU
Select Start Run

As the experiment results were completing, you could see an order of magnitude difference between having access to GPUs and having to train the model on CPU only.

Your results should be similar to:

The training time utilized for 0 GPU should be comparable to on-premises with no GPUs.

You can review the output of the python program, main.py, by selecting a Run id, then select Session.

Summary

Congratulations on completing the tutorial.

As you’ve now experienced, having access to a hybrid cloud solution allows the opportunity to leverage cloud resources only when you need them. In our experiments, the use of GPUs resulted in huge time savings, empowering users to spend their valuable time creating value instead of waiting for their model to train.