Environment variables#

Environment variables let you exclude sensitive data from the versioning system of your Orchest projects.

Environment variables are injected in your Pipeline Steps, and can be natively retrieved with your chosen language. For example, in Python:

import os
secret = os.environ["MY_VAR"]

You can define environment variables for Projects, Pipelines, and Jobs.

Pipeline variables overwrite Project variables. For example, if MY_VAR=PROJ_VALUE is defined at the Project level and MY_VAR=PIP_VALUE at the Pipeline level, the value of MY_VAR for an interactive pipeline runs is PIP_VALUE.

Changes to the PATH variable are ignored since they could break code execution.

Warning

🚨 Environment variables are persisted within Orchest. Make sure only authorized users can acess your instance. See how to setup authentication in the settings section.

Project environment variables#

Project environment variables are visible to all pipelines in that respective project (including pipelines in jobs). Access your project environment variables through project settings:

  1. Open the Projects view in the left menu pane.

  2. Click on gear icon (in the settings column) in the row representing the project of interest.

  3. Towards the top you will find the Project environment variables section.

  4. Set your variables.

  5. Click the Save button.

Pipeline environment variables#

Access your pipeline environment variables through the pipeline settings:

  1. Open a Pipeline via the Pipelines option in the left menu pane.

  2. Click on Settings in the top right corner.

  3. Click on the Environment variables tab.

  4. Set your variables.

  5. Click the Save button.

Job environment variables#

Job environment variables are initialized by merging the project and pipeline environment variables when a job is run. Add to/edit these variables before running the job. Every Pipeline run belonging to this job will include these environment variables.

Note

💡 Only recurring scheduled jobs can have their environment variables edited after they have started.

Environment variables inside Notebooks#

Environment variables are available in JupyterLab kernels too. Restart the interactive session to refresh the kernels’ environment variables.