Development workflow

Building Orchest

Since Orchest is a fully containerized application you will first have to build the containers.

# Ensure you've installed orchest
./orchest install

# It is also possible to specify certain flags, running it without
# any will build all containers in parallel. Due to Docker's
# layering system this should be rather quick.
scripts/build_container.sh

Incremental development

Make sure Orchest is already installed first. See the regular installation process for more details.

Orchest supports incremental development by starting Orchest with the --dev flag. This allows you to make code changes that are instantly reflected, without having to build the containers again.

Note

For incremental development to work in WSL2, Docker must be installed within the WSL2 environment itself.

# Before Orchest can be run in "dev" mode the front-end code has to
# be compiled
# (for the first time you may need to add the `--install` flag)

# First time setup
npm run setup
pnpm i

# Run the Vite dev server to for hot reloading. Note: This command
# does not finish.
pnpm run dev

# Run this command in a different terminal window.
./orchest start --dev

Note

The --dev flag affects the following services: orchest-webserver, auth-server, file-manager and orchest-api. For changes to other services you will have to run the build script again to rebuild the container (scripts/build_container.sh -i <service-name>) and restart Orchest (./orchest restart --dev) to make sure the newly built container is used.

With --dev the repository code from the filesystem is mounted (and thus adhering to git branches) to the appropriate paths in the Docker containers. This allows for active code changes being reflected inside the application. With --dev the Flask applications are run in development mode.

Tip

If the development mode hangs it’s likely that Vite has left a daemon running that is stuck (the giveaway is that a Vite start runs on ports other than 3000/3001). To fix the issue, run killall node and restart Vite.

Building the docs

Our docs are handled by Read the Docs with Sphinx and written in reStructuredText.

To build the docs, run:

cd docs

# First time setup
python3 -m pip install -r requirements.txt

# Build
make html

Best practices

When adding new code to the repository, try to stick to the following best practices (note that this list is a work in progress):

  • New endpoints, e.g. in the orchest-api or proxy in the orchest-webserver, should NOT end with trailing slashes. For example, go with /api/jobs (good) over /api/jobs/ (bad).

Before committing

Install all development dependencies using:

# if not run in prior development step
npm run setup
pnpm i

pre-commit install

Run formatters, linters and tests with:

pre-commit run
scripts/run_tests.sh