Skip to content

GitOps for Kubeflow Pipelines Schedules

Learn how to use GitOps to manage Kubeflow Pipelines, including pipeline definitions and pipeline schedules.

Overview

Initially, most users of Kubeflow Pipelines manually create and run workflows with the UI or Python SDK as this is the fastest way to get started. When the number of pipelines grows, it becomes increasingly difficult and error-prone to manage them manually; this is where GitOps comes in.

Reference Implementation

We provide a reference implementation for managing pipeline definitions and their schedules using GitOps in the deployKF/kubeflow-pipelines-gitops GitHub repo.

Check out the Reference Repository

The reference architecture is logically grouped into four steps:

Step Description
Step 1: Render Pipelines Render pipeline definitions into their static YAML representation.
Step 2: Run Pipelines Run the rendered pipelines ad-hoc.
Step 3: Schedule Pipelines Schedule the rendered pipelines.
Step 4: Automatic Reconciliation Automatically reconcile the schedule configs.
Last update: 2024-03-14
Created: 2023-08-16