Kubeflow

Kubeflow is a platform for running machine learning workflows on Kubernetes. Each stage of a machine learning workflow, such as training, serving, monitoring and logging, runs in containers on the Kubernetes cluster. This makes a distributed machine learning deployment easier to manage.

Kubeflow hides much of the work of managing a Kubernetes cluster. A machine learning practitioner can then use Kubernetes and the benefits of a microservices architecture without becoming a Kubernetes expert. Kubeflow started at Google in 2017 as an open-source way to run TensorFlow jobs on Kubernetes. It was modeled on the way Google ran TensorFlow internally with TensorFlow Extended. It has since grown to support many machine learning frameworks. Kubeflow joined the Cloud Native Computing Foundation as an incubating project in 2023 and graduated in 2026.

Working with Kubeflow

Kubeflow is made of subprojects. You can install each subproject on its own. You can also install them together as a Kubeflow distribution. The main subprojects are:

• Kubeflow Pipelines: builds and runs portable machine learning workflows from containerized steps. We cover it in the next part.
• Kubeflow Notebooks: runs web-based development environments, such as Jupyter notebooks, inside the cluster.
• Katib: tunes hyperparameters and searches for neural network architectures.
• Kubeflow Trainer: runs distributed training and fine-tuning jobs for frameworks such as PyTorch.
• Spark Operator: runs Apache Spark applications on Kubernetes.
• Kubeflow Hub: stores models and their metadata in a model registry and a model catalog.
• Central Dashboard: the web interface that links the Kubeflow components together.

The Community Distribution also installs KServe for model serving. It uses Istio for networking and Dex with OAuth2 Proxy for user login. See Installing Kubeflow for the full list of subprojects and the packaged distributions offered by vendors.

Installing Kubeflow

There are two main ways to install Kubeflow:

• Kubeflow Community Distribution: a set of Kustomize manifests maintained by the Kubeflow community in the kubeflow/community-distribution repository. It installs all the official components and the services they need. The project uses calendar versions and plans two releases a year. At the time of writing, the latest stable release is 26.03.1.
• Packaged distributions: versions of Kubeflow that vendors build, test and support for their own platforms.

Earlier versions of Kubeflow had a dedicated Google Cloud deployment that used the kfctl tool, ksonnet and Cloud Identity-Aware Proxy. The kfctl tool and ksonnet are no longer used. Today we install the Community Distribution on GKE with kustomize and kubectl.

If you only need Kubeflow Pipelines, you can install it on its own or run your pipelines on a managed service. The next part covers both options.

Set Up a Kubernetes Cluster on GKE

The Community Distribution recommends at least 16 GB of memory and 8 CPU cores for the full platform. Its Istio components need a small change on GKE. We make this change below. We create a GKE Standard cluster with two e2-standard-8 nodes. Each node has 8 vCPUs and 32 GB of memory.

bashCopy
# set the default project
gcloud config set project [PROJECT_ID]

# enable the kubernetes engine api
gcloud services enable container.googleapis.com

# create a gke standard cluster
gcloud container clusters create kubeflow-cluster \
    --location=us-central1-a \
    --machine-type=e2-standard-8 \
    --num-nodes=2

When the cluster is ready, gcloud configures kubectl to use it. Check the nodes:

bashCopy
# list the nodes of the cluster
kubectl get nodes

The output lists two nodes with the status Ready. The release notes of each Kubeflow release list the Kubernetes versions that the community tested. Check them against the version of your GKE cluster.

Install Kustomize

The installation uses the standalone kustomize tool. Follow the Kustomize installation guide for your operating system. On macOS with Homebrew, run:

bashCopy
brew install kustomize

Download the Kubeflow Manifests

bashCopy
# download the manifests at the 26.03.1 release
git clone --branch 26.03.1 https://github.com/kubeflow/community-distribution.git

# move into the repository
cd community-distribution

Adjust the Manifests for GKE

GKE mounts the folder where Istio normally installs its network plugin as read-only. The repository has a GKE overlay for Istio that uses a writable folder instead. Open example/kustomization.yaml and replace this line:

yamlCopy
- ../common/istio/istio-install/overlays/oauth2-proxy

with this line:

yamlCopy
- ../common/istio/istio-install/overlays/gke

You can also make the change with sed:

bashCopy
# switch to the gke overlay for istio
sed -i.bak 's|istio-install/overlays/oauth2-proxy|istio-install/overlays/gke|' example/kustomization.yaml

Deploy Kubeflow

The command below builds all the manifests and applies them to the cluster. Some resources depend on others that are not ready yet. The first attempts can therefore fail. The loop retries every 20 seconds until every resource applies.

bashCopy
# install all kubeflow components
while ! kustomize build example | kubectl apply --server-side --force-conflicts -f -; do echo "Retrying to apply resources"; sleep 20; done

The pods can take several minutes to start. Check them with:

bashCopy
# list the pods in all namespaces
kubectl get pods --all-namespaces

Kubeflow is ready when the pods in the cert-manager, istio-system, auth, oauth2-proxy, knative-serving, kubeflow and kubeflow-user-example-com namespaces show the status Running.

Open the Kubeflow Central Dashboard

Forward a local port to the Istio ingress gateway:

bashCopy
# forward local port 8080 to the istio ingress gateway
kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80

Open http://localhost:8080 in a browser. The Dex login screen appears. Log in with the default email user@example.com and the default password 12341234. Change this password before you use Kubeflow for anything beyond a test. The Community Distribution README explains how to change it, how to connect your own identity provider and how to expose the dashboard through a load balancer or an ingress.

After you log in, the Central Dashboard shows links to Pipelines, Notebooks, Katib experiments, model serving and the other components.

Delete Resources

See the end of Deploying an End-to-End Machine Learning Solution on Kubeflow Pipelines to delete the billable Google Cloud resources.