Code management: scc

scc is a Python library with a set of utility commands used for code management and used in the OME Continuous integration. More information can be found in the Python package page or in the source code page.

If you find a bug or if you want an additional feature to be implemented, please open an issue.

Installation

The scc tools are a set of Python based utility programs. The tools suite can be installed using pip:

$ pip install -U scc

This command will install and/or upgrade the PyGithub and yaclifw package dependencies. If the version of Python installed is older than 2.7, this may also install the argparse package.

Github connection

Most of the scc commands instantiate a Github connection using the PyGithub package. GitHub strongly recommends to turn on two-factor authentification (2FA), see About Two-Factor Authentication for more details. If 2FA is activated, the only way to use scc commands creating a GitHub connection is to create an OAuth token, see Creating an access token for command-line use for details on how to create Personal Access Tokens via the GitHub interface. This token can then be stored in the global Git configuration file:

git config --global github.token REPLACE_BY_PERSONAL_ACCESS_TOKEN

Unless the --token option is passed to the scc command, the command first looks for the github.token specified in the git config file and, if found, uses this token to connect to GitHub:

$ scc merge master --info -v
2013-01-16 22:03:49,633 [  scc.config] DEBUG Found github.token
...

If no token is found, the command looks for a github.user in the git config file and, if found, uses this username to connect to Github:

$ scc merge master --info -v
2013-01-16 22:06:00,256 [  scc.config] DEBUG Found github.user
Enter password for https://github.com/sbesson:

Note

The password to be entered here is the GitHub password. Connecting using the GitHub username/password is NOT possible if 2FA has been activated.

Finally, if no token or user is found, both the GitHub username and password are queried at the prompt:

$ scc merge master --info -v
# github.token and github.user not found.
# See `scc token` for simpifying use.
Username or token: sbesson
Enter password for https://github.com/sbesson:

scc merge

Merge all the PRs based on specified branch matching the input filters including all submodules.

Description

Filters of different types can be specified and combined to include and exclude a set of PRs in the merge command. Each filter needs to be formatted as key:value. If no key but only a value is specified, it is assumed the filter is a label filter (see below). These filters can be passed to an scc merge --include or scc merge --exclude option.

The available filter types are described below:

Label filters can be specified using the label key i.e. label:<LABEL>. This filter type will match a Pull Request if one of the following conditions is met:
1. a label named <LABEL> is applied to the Pull Request
2. the Pull Request description contains a line starting with --<LABEL>
3. one of the Pull Request comments contains a line starting with --<LABEL>. Note this comment needs to be written by one of the public members of the organization owning the upstream repository.
User filters can be specified using the user key i.e. user:<USER>. This filter type will select a Pull Request if it has been opened by the user USER. Additionally, two special user values are allowed:
1. the #org value will match all PRs opened by public members of the organization of the upstream repository
2. the #all value will match all PRs opened by any user
PR filters can be specified using the pr key i.e. pr:<NUMBER>. This will select Pull Requests whose ID matches the input number. The form #number is also recognized as a PR filter. For repositories containing submodules, it is possible to filter submodule PRs using user/repo#number.

Arguments

The first argument is the name of the base branch of origin, e.g.:

$ scc merge develop

--comment

Add a comment to the PR if there is a conflict while merging the PR

$ scc merge develop --comment

--default <filterset>, -D <filterset>

Specify the default set of filters to use

Three filter sets are currently implemented: none, org and all. The none filter set has no preset filter. The org filter set uses user:#org and label:include as the default include filters and label:exclude and label:breaking as the default exclude filters. The all filter set uses user:#all as the default include filters.

Default: org

--exclude <filter>, -E <filter>

Exclude PR by filter (see filter semantics above):

$ scc merge develop -E label:l1 -E user:u1 -E #45 -E org/repo#40

--include <filter>, -I <filter>

Include PR by filter (see filter semantics above):

$ scc merge develop -I label:l1 -I user:u1 -I #45 -I org/repo#40

--check-commit-status <status>, -S <status>

Exclude PR based on the status of the last commit

Three options are currently implemented: none, no-error and success-only. By default (none), the status of the last commit on the PR is not taken into account. To include PRs which have a successful status only, e.g. PRs where the Travis build is green, use the success-only option:

$ scc merge develop -S success-only

To exclude all PRs with an error or failure status, use the no-error option:

$ scc merge develop -S no-error

--info

Display the candidate PRs to merge but do not merge them

$ scc merge develop --info

--push <branchname>

Push the locally merged branch to Github

$ scc merge develop --push my-merged-branch

--reset

Recursively reset each repository to the HEAD of the base branch

$ scc merge develop --reset

--shallow

Merge the PRs for the top-level directory only, excluding submodules:

$ scc merge develop --shallow

--remote <remote>

Specify the name of the remote to use as the origin. Default: origin:

$ scc merge develop --remote gh

As a concrete example, the first step of a merge job is calling the following merge command:

$ scc merge master --no-ask --reset --comment --push merge_ci

Use cases

The basic command will use the default filters and merge all PRs opened against master by any public members of the organization, include any PR labeled as include and exclude any PR labeled as breaking or exclude:

$ scc merge master

The following command overrides the default set of filters and will only merge PRs opened against master labeled as my_label:

$ scc merge master -Dnone -Ilabel:my_label

The following command overrides the default set of filters and will merge all PRs opened against master by public members of the organization, include any PR labeled with my_label and exclude any PR labeled as exclude:

$ scc merge master -Dnone -Iuser#org -Ilabel:my_label -Elabel:exclude

Changed in version 0.3.0: Added default values for --include and --exclude options.

Changed in version 0.3.8: Added --shallow and --remote options.

Changed in version 0.4.0: Added --check-commit-status option.

scc travis-merge

Merge PRs in a Travis environment, using the PR comments to generate the merge filters.

$ scc travis-merge

This command internally defines all the filter options exposed in scc merge.

The target branch is read from the base of the PR, the scc merge --default option is set to none meaning no PR is merged by default and no default scc merge --exclude option is defined.

The scc merge --include filter is determined by parsing all the PR comments lines starting with --depends-on.

To include a PR from the same GitHub repository, use the PR number prepended by #. For instance, to include PR 67 in the Travis build, add a comment line starting with --depends-on #67 to the PR.

To include a PR from a submodule, use the PR number prepended by submodule_user/submodule_name#. For instance, to include PR 60 of bioformats in the Travis build, add a comment line starting with --depends-on openmicroscopy/bioformats#60 to the openmicroscopy PR.

Note

The scc travis-merge command works solely for Pull Requests’ Travis builds.

scc update-submodules

Update the pointer of all submodules based on specified branch.

The first argument is the name of the base branch of origin, e.g.:

$ scc update-submodules develop

--push <branchname>

Push the locally merged branch to Github and open a PR against the base branch:

$ scc merge develop --push submodules_branch

--no-pr

Combined with --push option, push the locally merged branch to Github but skip PR opening:

$ scc merge develop --push submodules_branch --no-pr

--remote <remote>

Specify the name of the remote to use as the origin (default: origin):

$ scc update-submodules develop --remote gh

scc rebase

Rebase a PR (open or closed) onto another branch and open a new PR.

The first argument is the number of the PR to rebase and the second argument is the name of the branch onto which the PR should be rebased:

$ scc rebase 142 develop

Assuming the head branch used to open the PR 142 was called branch_142, this command will rebase the tip of branch_142 onto origin/develop, create a new local branch called rebased/develop/branch_142, push this branch to Github and open a new PR. Assuming the command opens PR 150, to facilitate the integration with scc check-prs, a --rebased-to #150 comment is added to PR 142 and a --rebased-from #142 comment is added to the PR 150. Finally, the command will switch back to the original branch prior to rebasing and delete the local rebased/develop/branch_142.

Note

By default, scc rebase uses the branches of the origin remote to rebase the PR. To specify another remote, use the --remote option.

--no-pr

Skip the opening of the PR

$ scc rebase 142 develop --no-pr

--no-delete

Do not delete the local rebased branch

$ scc rebase 142 develop --no-delete

--remote <remote>

Specify the name of the remote to use for the rebase (default: origin)

$ scc rebase 142 develop --remote snoopycrimecop

--continue

Re-run the command after manually fixing conflicts

If scc rebase fails due to conflict during the rebase, you will end up in a detached HEAD state.

If you want to continue the rebase operation, you will need to manually fix the conflicts:

# fix files locally
$ git add conflicting_files # add conflicting files
$ git rebase --continue

This conflict solving operation may need to be repeated multiple times until the branch is fully rebased.

Once all the conflicts are resolved, call scc rebase with the --continue option:

$ scc rebase --continue 142 develop

Depending on the input options, this command will perform all the steps of the rebase command (Github pushing, PR opening) skipping the rebase part.

Alternatively, you can abort the rebase and switch to your previous branch:

$ git rebase --abort
$ git checkout old_branch

Changed in version 0.3.10: Automatically add --rebased-to and --rebased-from comments to the source and target PRs.

scc check-prs

Compare two development branches and check that PRs merged in one branch have been merged to the other.

The basic workflow of the scc check-prs command is the following:

list all first-parent merge commits for each branch including git notes referenced as see_also/other_branch where other_branch is the name of the branch to check against.
exclude all merge commits with a note containing either “See gh-” or “n/a”
for each remaining merge commit, parse the PR number and look into the PR body/comments for lines starting with --rebased-to, --rebased-from or --no-rebase.

Additionally, for each line of each PR starting with --rebased-to or --rebased-from, the existence of a matching line is checked in the corresponding source/target PR. For instance, if PR 70 has a --rebased-from #67 line and a --rebased-from #66 line, then both PRs 66 and 67 should have a --rebased-to #70 line.

This command requires two positional arguments corresponding to the name of the branch of origin to compare:

$ scc check-prs dev_4_4 develop

--shallow

Check PRs in the top-level directory only, excluding submodules:

$ scc check-prs dev_4_4 develop --shallow

--remote <remote>

Specify the name of the remote to use as the origin (default: origin):

$ scc check-prs dev_4_4 develop --remote gh

--no-check

Do not check links between rebased comments:

$ scc check-prs dev_4_4 develop --no-check

New in version 0.3.10: Added support for body/comment parsing and --rebased-to/from linkcheck

Changed in version 0.4.0: Improved command output and added support for submodule processing

Changed in version 0.5.0: Renamed command

scc version

Return the version of the scc tools:

$ scc version
0.3.0

New in version 0.2.0.

scc deploy

Deploy a website update using file symlink replacement:

$ scc deploy folder

The goal of this command is to enable overwriting of deployed doc content and allow for “hot-swapping” content served by Apache without downtime and HTTP 404s.

--init

Prepare folder for symlink replacement. Should only be run once

$ scc deploy folder --init

New in version 0.3.1.

The hudson jobs ending with release-docs and OMERO-docs-internal deploy the documentation artifacts to necromancer. The target directory (sphinx-docs) is controlled by the hudson:hudson user, so all file system operations are allowed. Each job has the target directory configured in the SSH publisher target directory property. After deployment has happened to a temporary directory, a series of symlink moves happens making sure that the symlink points to the updated content.

scc check-status

Check the status of the Github API:

$ scc check-status && echo "Passing"
Passing

-n <N>

Display N last status messages from Github API history:

$ scc check-status -n 4
2013-11-04 13:40:48 (minor) We're investigating an increase in error responses from the API.
2013-11-04 14:33:55 (good) Everything operating normally.
2013-11-05 12:59:50 (minor) We're investigating reports of an increase in 502s from the GitHub API.
2013-11-05 13:07:15 (good) Everything operating normally.

New in version 0.4.0.