To pass data between the different pipeline steps, across different languages, we make use of the Apache Arrow project. The Orchest SDK provides a convenience wrapper of the project to be used within Orchest.
See Data passing to get started directly.
Installing additional packages¶
To install additional packages or to run other terminal commands inside the base image, we support custom Images. We essentially create a new image by running your script inside the selected base image.
- Simply go to Images in the left menu pane.
- Select the base image. This image will be extended with your custom script.
- Click the “+” sign to add a commit to the base image. The commit represents the changes of your script.
- Choose a Commit name.
- Install additional packages, e.g.
pip install tensorflowor
sudo apt install vim.
Do not install packages by running
!pip install <package-name> inside your
Jupyter Notebook. This causes the package to be installed every time you run the pipeline
step. It is not saved in the environment as containers are stateless!
Before we get into experiments, it is good to first cover the notion of parameterizing your pipeline steps.
- Open a pipeline via the Pipelines option in the left menu pane.
- Click on a pipeline step to open its Properties.
- Towards the bottom you will find the Parameters section.
- Input some JSON like
- Make sure to press the black Save button towards the top of your screen.
You can now access the value of those parameters (and even update them) from within the script of the respective pipeline step. See Parameters to get started on using parameters inside your scripts.
Now that we have covered how to parameterize your pipeline (steps) we will introduce experiments. With experiments you get to try out all your modeling ideas by iterating over different parameter values. For now you can think of it as a grid search. To start an experiment
- Make sure you have defined some parameters or you will only be able to schedule the pipeline as is.
- Click on Experiments in the left menu pane.
- Click the “+” sign to configure your experiment.
- Choose an “Experiment name” and the “Pipeline” you want to run the experiment for.
- Your default set of parameters are pre-loaded. By clicking on the values a JSON editor opens, allowing you to add additional values you would like to try out.
- If you would like to schedule the experiment to run at a specific time have a look at Scheduling. In case you don’t want your grid search to run every combination of your parameter values, you can deselect them through the Pipeline runs option.
- Press Run experiment.
To inspect the result of your experiment, simply click on the experiment you just created, choose a specific pipeline run (the one you want to inspect) and open View pipeline. The pipeline is now opened in Read only mode giving you the opportunity to check the logs or to open the HTML version of you notebooks.
It is common for data to be stored elsewhere and to connect to those data sources from within your scripts. To make sure that all the secrets can be managed seperately, they are stored in a central place.
Get started with data sources by configuring one through the Data sources option in the left pane menu.
- Click on Data sources in the left menu pane.
- Click on the “+” sign to add a data source.
- Specify a “Name” and “Data source type”.
- Put in the “Connection details” to connect to the data source. For example for a MySQL database, you’d have to specify the “Host”, “Database name”, “Username” and “Password”.
- Lastly, press Save.
Next, see Data sources in the SDK section to get started with data sources from within your scripts.
For full reproducibility of your experiments, Orchest creates a snapshot of your active pipeline directory. Therefore it is recommended to never write large amounts of data to your pipeline directory but to use the Filesystem directory data source type instead. It is nothing more than a special path that gets mounted for the purpose of storing large amounts of data.
Tips and tricks¶
- Hold down
<Space>inside the pipeline editor to drag the canvas (similar to design tools such as Sketch).
- On your host machine, in the terminal, run
docker ps -f network=orchestto see all the containers that Orchest is running.