Multiple Repository Management

Introduction

West includes a set of commands for working with projects composed of multiple Git repositories installed under a common parent directory (a west installation), similar to how Git Submodules and Google’s repo work.

The rest of this page introduces these multi-repo concepts and gives an overview of the associated west commands, along with an example workflow. See Zephyr issue #6770 for additional background and discussion.

Note

The multi-repo commands are meant to augment Git in minor ways for multi-repo work, not replace it. For tasks that aren’t multi-repo-related, use plain Git commands. This page explains what the west multi-repo commands do behind the scenes.

A west installation is the result of running the west init command to either create a new installation or convert a standalone zephyr repository into a multi-repo installation. When using upstream Zephyr, it looks like this:

└── zephyrproject/
    ├── .west/
    │   ├── config
    │   └── west/
    ├── zephyr/
    │   └── west.yml
    ├── a-project
    └── ...

Above, zephyrproject is the name of the west installation’s common parent directory, and a-project is another project managed by west in the installation. The file .west/config is the installation’s west configuration file. The directory .west/west is a clone of the west repository itself; more details on why that is needed are given below.

Every west installation contains a manifest repository, which is a Git repository containing a file named west.yml. This file, along with what’s in .west, controls the installation’s behavior. In the above example, zephyr is the manifest repository. However, as you’ll see below, any repository in an installation can be the manifest repository; it doesn’t have to be zephyr. However, every installation has exactly one manifest repository; its location is specified in .west/config. Alongside the manifest repository are projects, which are Git repositories specified by west.yml.

Requirements

Although the motivation behind splitting the Zephyr codebase into multiple repositories is outside of the scope of this page, the fundamental requirements along with a clear justification of the choice not to use existing tools and instead develop a new one, do belong here. At the most fundamental level, the requirements for a transition to multiple repositories in the Zephyr Project are:

  • R1: Keep externally maintained code outside of the main repository
  • R2: Provide a tool that both Zephyr users and distributors can make use of to benefit from and extend the functionality above
  • R3: Allow overriding or removing repositories without having to make changes to the zephyr repository
  • R4: Support both continuous tracking and commit-based (bisectable) project updating

Topologies supported

The requirements above lead us to the three main source code organization topologies that we intend to address with west:

  • T1: Star topology with zephyr as the manifest repository:
    • The zephyr repository acts as the central repository and includes a complete list of projects in itself
    • Default (upstream) configuration
    • Analogy with existing mechanisms: Git submodules with zephyr as the superproject
  • T2: Star topology with an application repository as the manifest repository
    • The application repository acts as the central repository and includes a complete list of projects in itself
    • Useful for downstream distributions focused in a single application repository
    • Analogy with existing mechanisms: Git submodules with the application as the superproject, zephyr as a submodule
  • T3: Forest topology with a set of trees all at the same level
    • A dedicated manifest repository contains only a list of repositories
    • Useful for downstream distributions that track the latest HEAD on all repositories
    • Analogy with existing mechanisms: Google repo-based distribution

Rationale for a custom tool

During the different stages of design and development for west, using already established and proven multi-repo technologies was considered multiple times. After careful analysis, no existing tool or mechanism was found suitable for Zephyr’s use case set and requirements. The following two tools were examined in detail:

  • Google repo
    • The tool is Python 2 only
    • Does not play well with Windows
    • It is poorly documented and maintained
    • It does not fully support a set of bisectable repositories without additional intervention (R4)
  • Git submodules
    • Does not fully support R1, since the externally maintained repositories would still need to be inside the main zephyr Git tree
    • Does not support R3. Downstream copies would need to either delete or replace submodule definitions
    • Does not support continuous tracking of the latest HEAD in external repositories (R4)
    • Requires hardcoding of the paths/locations of the external repositories

Finally, please see History and Motivation for the motivations behind using a single tool for both multi-repository management as well as building, debugging and flashing.

Structure

West structure

West is currently downloaded and installed on a system in the following stages:

  • Bootstrapper: Installed using pip, implements west init
  • Installation: Installed using west init, implements built-in multi-repo commands
  • Extension commands: Installed using west update, parses the manifest in the west installation for additional commands

Note

The separation between the “bootstrapper” and “installation” pieces is inspired by the Google Repo tool, but it’s a temporary structure for convenience. In the future, the two will be merged. This will lessen the complexity of the design and make it easier to write and debug extension commands.

Repository structure

Beyond west itself, the actual code repositories that west works with are the following:

  • Manifest repository: Cloned by west init, and managed afterward with Git by the user. Contains the list of projects in the manifest file west.yml. In the case of upstream Zephyr, the zephyr repository is the manifest repository.
  • Projects: Cloned and managed by west update. Listed in the manifest file.

Bootstrapper

The bootstrapper module is distributed using PyPI and installed using pip. A launcher named west is placed by pip in the user’s PATH. This the only entry point to west. It implements a single command: west init. This command needs to be run first to use the rest of functionality included in west, by creating a west installation. The command west init does the following:

  • Clone west itself in a .west/west folder in the installation
  • Clone the manifest repository in the folder specified by the manifest file’s (west.yml) self.path section. Additional information on the manifest can be found in the Multiple Repository Management section)

Once west init has been run, the bootstrapper will delegate the handling of any west commands other than init to the cloned west installation.

This means that there is a single bootstrapper instance installed at any time (unless you use virtual environments), which can then be used to initialize as many installations as needed.

Installation

A west installation, as describe above, contains a clone of the west repository in .west/west. The clone of west in the installation is where the built-in multi-repo command implementations are currently provided.

When running a west command, the bootstrapper delegates handling for all commands except init to the .west/west repository in the current installation.

Extension Commands

The west manifest file (more about which below) allows any repository in the installation to provide additional west commands (the flash and debug commands use this mechanism; they are defined in the west repository). Every repository which includes extension commands must provide a YAML file which configures the names of the commands, the Python files that implement them, and other metadata.

The locations of these YAML files throughout the installation are specified in the manifest, which you’ll read about next.

Model

Manifest

A manifest is a YAML file that, at a minimum, gives the URL and revision for each project repository in the multi-repo (each project). The manifest is stored in the manifest repository and is named west.yml.

The format of the west manifest is described by a pykwalify schema, manifest-schema.yml.

A west-based Zephyr installation has a special manifest repository. This repository contains the west manifest file named west.yml. The west manifest contains:

  • A list of the projects (Git repositories) in the installation and metadata about them (where to clone them from, what revision to check out, etc.). Externally maintained projects (vendor HALs, crypto libraries, etc) will be moved into their own repositories as part of the transition to multi-repository, and will be managed using this list.
  • Metadata about the manifest repository, such as what path to clone it into in the Zephyr installation.
  • Optionally, a description of how to clone west itself (this allows users to fork west itself, should that be necessary).
  • Additionally both the projects and the metadata about the manifest repository can optionally include information about additional west commands (extensions) that are contained inside the Git repositories.

Repositories

There are therefore three types of repositories that exist in a west-based Zephyr installation:

  • West repository: This is cloned by west init and placed in .west/west
  • Manifest repository: This is the repository that contains the west.yml manifest file described above which lists all the projects. The manifest repository can either contain only the manifest itself or also have actual code in it. In the case of the upstream Zephyr Project, the manifest repository is the zephyr repository, which contains all zephyr source code except for externally maintained projects, which are listed in the west.yml manifest file. It is the user’s responsibility to update this repository using Git.
  • Project repositories: In the context of west, projects are source code repositories that are listed in the manifest file, west.yml contained inside the manifest repository. West manages projects, updating them according to the revisions present in the manifest file.

West’s view of multirepo history looks like this example (though some parts of the example are specific to upstream Zephyr’s use of west):

West multi-repo history

West multi-repo history

The history of the manifest repository is the line of Git commits which is “floating” on top of a plane (parent commits point to child commits using solid arrows.) The plane contains the Git commit history of the projects, with each project’s history boxed in by a rectangle.

The commits in the manifest repository (again, for upstream Zephyr this is the zephyr repository itself) each have a manifest file. The manifest file in each zephyr commit gives the corresponding commits which it expects in the other projects. These are shown using dotted line arrows in the diagram.

Notice a few important details about the above picture:

  • Other projects can stay at the same versions between two zephyr commits (like P2 does between zephyr commits A B, and both P1 and P3 do in F G).
  • Other projects can move forward in history between two zephyr commits (P3 from A B).
  • A project can also move backwards in its history as zephyr moves forward (like P3 from zephyr commits C D). One use for this is to “revert” a regression by moving the other project to a version before it was introduced.
  • Not all zephyr manifests have the same other projects (P2 is not a part of the installation at zephyr commits F and G).
  • Two zephyr commits can have the same external commits (like F and G).
  • Not all commits in some projects are associated with a zephyr commit (P3 “jumps” multiple commits in its history between zephyr commits B C).
  • Every zephyr commit’s manifest refers to exactly one version in each of the other projects it cares about.

The manifest-rev branch

West creates a branch named manifest-rev in each project, pointing to the project’s manifest revision (or, more specifically, to the commit the revision resolves to). The manifest-rev branch is updated whenever project data is fetched (the command overview below explains which commands fetch project data).

All work branches created using west track the manifest-rev branch. Several multi-repo commands also use manifest-rev as a reference for the upstream revision (as of the most recent fetch).

Note

manifest-rev is a normal Git branch, and is only treated specially by name. If you delete or otherwise modify it, it will be recreated/reset when upstream data is next fetched by west, as if through git reset --hard (though git update-ref is used internally).

Since manifest-rev represents the upstream revision as of the most recent fetch, it is normally a bad idea to modify it.

manifest-rev was added to allow branches to track SHA revisions, and to give a consistent reference for the upstream revision regardless of how the manifest changes over time.

Note

West does not create a manifest-rev branch in the manifest repository, since west does not manage the manifest repository’s branches or revisions.

Command overview

This section gives a quick overview of the multi-repo commands, split up by functionality. Some commands loosely mimic the corresponding Git command, but in a multi-repo context (west diff shows local changes on all repositories).

Passing no projects to commands that accept a list of projects usually means to run the command for all projects listed in the manifest.

Note

For the most up-to-date documentation, see the command help texts (e.g. west diff --help). Only the most important flags are mentioned here.

Cloning and updating projects

After running west init to initialize west (e.g., with the default Zephyr manifest), the following commands will clone/update projects.

Note

To implement self-updates, west init also clones a repository with the west source code, which is updated whenever the projects are. The west command is implemented as a thin wrapper that calls through to the code in the cloned repository. The wrapper itself only implements the west init command.

This is the same model used by Google’s repo tool.

  • west init [-l] [-m URL] [--mr REVISION] [PATH]: Initializes a west installation.

    This command can be invoked in two distinct ways. If you already have a local copy or clone of the manifest repository, you can use the -l switch to instruct west to initialize an installation around the existing clone, without modifying it. For example, west init -l path/to/zephyr is useful if you already have cloned the zephyr repository in the past using Git and now want to initialize a west installation around it. If you however want to initialize an installation directly from the remote repository, you have the option to specify its URL using the -m switch and/or its revision with the --mr one. For example, invoking west with: west init -m https://github.com/zephyrproject-rtos/zephyr --mr v1.15.0 would clone the upstream official zephyr repository at the tagged release v1.15.0.

  • west update [PROJECT ...]: Clones or updates the specified projects (default: all projects).

    This command will parse the manifest file (west.yml) in the manifest repository, clone all project repositories that are not already present locally and finally update all projects to the revision specified in the manifest file. An initial branch named after the project’s manifest revision is created in each cloned project repository. The names of branch and tag revisions are used as-is. For qualified refs like refs/heads/foo, the last component (foo) is used. For SHA revisions, a detached HEAD is checked out.

Note

West uses git checkout to switch each project to the revision specified in the manifest repository. This is typically a safe operation that will not modify any branches or staged work you might have.

Miscellaneous commands

These commands perform miscellaneous functions.

  • west list: Lists project information from the manifest (URL, revision, path, etc.), along with other manifest-related information.

  • west diff [PROJECT ...] [ARGUMENT ...]: Runs a multi-repo git diff for the specified projects (default: all cloned projects).

    Extra arguments are passed as-is to git diff.

  • west status [PROJECT ...] [ARGUMENT ...]: Like west diff, for running git status.

  • west forall -c COMMAND [PROJECT ...]: Runs the shell command COMMAND within the top-level repository directory of each of the specified projects (default: all cloned projects).

    If COMMAND consists of more than one word, it must be quoted to prevent it from being split up by the shell.

    Note that west forall can be used to run any command, not just Git commands. To run a Git command, do west forall -c 'git ...'.

  • west selfupdate: Updates the west repository.

    Normally, the west repository is updated automatically whenever a command that fetches upstream data is run (this behavior can be suppressed for the duration of a single command by passing --no-update).