A Singapore Government Agency Website

Known issues after migrating to GitLab-hosted runners

When migrating from CStack runners (Kubernetes executor) to GitLab-hosted runners (Docker Autoscaler executor), you may encounter permission or environment-related errors. This section explains the root causes and how to fix them.

Summary of Common Issues

Issue Common cause Recommended fix
Permission denied on artifacts/cache Files owned by root Run as root or fix file ownership
Permission denied in .cache or .local Unset or mismatched HOME Set HOME=/tmp or other writable path
$HOME or similar variables expand incorrectly Evaluated before container starts Export variables dynamically in before_script

Q1. Why do I get “Permission denied” errors on files or directories created by my CI jobs?

Symptom: Jobs fail with errors like:

Permission denied: cannot write to /builds/project/... or /cache/...

Cause: GitLab-hosted runners execute your job container as the root user by default, even if your job image specifies a non-root USER. As a result, any artefacts or cache files extracted by the runner are owned by root.

If your job image runs its processes as a non-root user, that user lacks permission to modify these root-owned files or directories.

Fix:

  • Use a job image that runs as root. This way, your job can read, write, and fix file permissions without errors.

Symptom: Jobs fail with errors referencing paths such as:

Permission denied: /root/.cache/pip

or

Permission denied: /root/.local/share

Cause: CStack runners run jobs as non-root with a defined home directory. GitLab-hosted runners run as root, and environment variables like HOME may differ or be unset.

As a result, jobs that expect a writable $HOME (for .cache, .local, or other directories) may fail when these paths point to unexpected or restricted locations.

Fix:

  • Explicitly define writable environment paths in your .gitlab-ci.yml, for example:

    ```yaml variables: HOME: “/tmp” before_script:

  • mkdir -p “$HOME/.cache” ```
  • Ensure the directory exists and has write access for the user running the job.

Q3. Why do GitLab job variables referencing environment variables like $HOME behave differently?

Symptom: A job variable referencing another environment variable resolves incorrectly. For example:

variables:
  DOCKER_AUTH_CONFIG_PATH: "$HOME/.docker/config.json"

may expand to:

/.docker/config.json

instead of:

/root/.docker/config.json

Cause: How GitLab resolves variables differs between runner types:

Runner type Evaluation time Environment used
CStack Runner (Kubernetes) Runtime (inside container) Container’s environment variables, including $HOME
GitLab Hosted Runner (Docker Autoscaler) Before container starts Runner host environment (may lack $HOME)

Thus, $HOME and similar variables may not resolve as expected on GitLab-hosted runners.

Fix:

  • Avoid referencing runtime variables like $HOME in the variables: block. Instead, export them dynamically:

    ```yaml before_script:

  • export DOCKER_AUTH_CONFIG_PATH=”$HOME/.docker/config.json” ```
  • Ensure the environment variables are defined inside your Docker image (ENV HOME=/root in the Dockerfile).

  • Alternatively, explicitly set environment variables in .gitlab-ci.yml:

    variables:
    HOME: "/tmp"
    DOCKER_AUTH_CONFIG_PATH: "/tmp/.docker/config.json"
Chat with me!
Bot Launcher