Kubeflow Pipelines

Kubeflow Pipelines (KFP) is a platform for building and deploying machine learning workflows that run in containers on Kubernetes. With Kubeflow Pipelines, you can build production machine learning pipelines without handling the low-level details of the Kubernetes cluster.

Kubeflow Pipelines is a core component of Kubeflow. It is installed when you install the Kubeflow Community Distribution, as we did in the previous part. You can also install it on its own.

Components of Kubeflow Pipelines

A pipeline describes a machine learning workflow. Each step of the pipeline is a component, a self-contained piece of code that runs in its own container. The pipeline connects the components into a graph and passes the outputs of one step to the inputs of the next. A pipeline also takes input parameters for each run.

The Kubeflow Pipelines platform consists of:

• A Python SDK for writing components and pipelines.
• A compiler that turns a pipeline into a YAML file in an intermediate representation (IR). Any backend that follows the KFP specification can run this file.
• A backend that schedules and runs the steps of each pipeline on Kubernetes.
• A user interface (UI) for managing and tracking pipelines, experiments and runs.

See the Kubeflow Pipelines overview for more detail.

Where Pipelines Run

A compiled pipeline can run in three places:

1. Kubeflow: the Pipelines component of a full Kubeflow installation.
2. Kubeflow Pipelines on its own: the open-source backend installed on any Kubernetes cluster, such as a GKE cluster.
3. Agent Platform Pipelines: a serverless service on Google Cloud that runs pipelines written with the KFP SDK. It was called Vertex AI Pipelines until April 2026. Google then introduced Gemini Enterprise Agent Platform as the successor to Vertex AI. Google runs the infrastructure for this service. You do not need to create a cluster. See the introduction to Agent Platform Pipelines.

In the next part, we write one pipeline and run it on both the open-source backend and Agent Platform Pipelines.

Install Kubeflow Pipelines on GKE

If you installed Kubeflow in the previous part, Kubeflow Pipelines is already running. Open it from the Pipelines link in the Central Dashboard and skip to the next section.

Otherwise, install Kubeflow Pipelines on its own. This is the development setup from the installation guide. It has no user login. Use it only for experiments. First, create a GKE Standard cluster:

bashCopy
# create a gke standard cluster
gcloud container clusters create kfp-cluster \
    --location=us-central1-a \
    --machine-type=e2-standard-4 \
    --num-nodes=2

Then install Kubeflow Pipelines. At the time of writing, the latest release is 2.17.2.

bashCopy
# set the kubeflow pipelines version
export PIPELINE_VERSION=2.17.2

# install the cluster-wide resources
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION"

# wait for the custom resource definitions to be ready
kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io

# install kubeflow pipelines
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/env/dev?ref=$PIPELINE_VERSION"

The deployment takes about 3 minutes. Then forward a local port to the UI service:

bashCopy
# forward local port 8080 to the kubeflow pipelines ui
kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80

Open http://localhost:8080 in a browser to see the Kubeflow Pipelines UI.

Executing a Sample Pipeline

Kubeflow Pipelines comes with two tutorial pipelines. We run the one named [Tutorial] DSL - Control structures. It shows conditional steps and an exit handler. The pipeline fails at random on purpose to show that the exit handler runs even when a step fails.

1. Click Pipelines in the side menu. The list shows the two tutorial pipelines.
2. Click [Tutorial] DSL - Control structures to see the graph of its steps.
3. Click Runs in the side menu. Then click Create run.
4. Select the pipeline and give the run a name. Choose an experiment or create a new one. An experiment groups related runs.
5. Click Start.
6. Click the name of the run to follow its progress. The graph shows the status of each step. Click a step to see its inputs, outputs and logs.

Delete Resources

See the end of Deploying an End-to-End Machine Learning Solution on Kubeflow Pipelines to delete the billable Google Cloud resources.