wiki:Tutorials/Advanced/Hosting Finroc Repositories

Hosting Finroc Repositories

At latest when you work in a team on Finroc code, you might want to host the repositories you specifically work with on your own server in you e.g. workgroup or company. That typically includes managing the repositories, access rights for users and group, updates from finroc.org, consistency checks and deployment via component lists that can be used by finroc_get or finroc_search.

This text is a step-by-step manual how to install an environment similar to the one at finroc.org. You might have to adapt it to your needs, e.g. do not want to use Trac for repository management or the Apache Webserver for repository access.

Install Dependencies

First, some dependencies must be installed. This guide was written on base of a server running Debian 7.0 (Wheezy). The correct version of astyle can always be downloaded as Debian package from our server after including our repository into apt's sources. Otherwise, finroc_code_formatter will tell you about the currently used version.

~# cat > /etc/apt/sources.list.d/finroc.org.list <<EOF
> deb     http://packages.finroc.org/debian wheezy main
> deb-src http://packages.finroc.org/debian wheezy main
> EOF
~# apt-get update
~# apt-get install -y --force-yes finroc-archive-keyring
~# apt-get update
~# apt-get install astyle mercurial libterm-readkey-perl libxml-simple-perl
~# cat > /etc/mercurial/hgrc.d/finroc-hostfingerprints.rc <<EOF
> [hostfingerprints]
> finroc.org = AD:C4:C9:99:88:71:EF:BD:75:09:1A:AC:89:06:7A:8F:7B:F4:15:AD
> EOF

Apache Webserver

For basic setup of the Apache Webserver please see its documentation. For our needs, some additional modules must be installed and activated to make the configuration shown below work:

~# apt-get install libapache2-mod-macro libapache2-mod-wsgi libapache2-svn
~# a2enmod authn_anon authz_svn macro

Trac

Please refer to the Trac documentation how to install it on your server. You will need at least version 0.12 with multi-repository-support. Also get the Mercurial-Plugin and our Repository-Manager plugin:

~# hg clone https://finroc.org/hg/repositorymanagerplugin
~# cd repositorymanagerplugin
~/repositorymanagerplugin# python setup.py install

Prepare the Environment

You might want to create a user for repository related things on the server. However, as much work can be done via the webserver, also its user www-data could be used instead. If a specific user for source code management is created, it must be member of the group www-data to share its belongings after changing the according modes on files and directories:

~# useradd scm_admin --system --gid www-data --home /srv/scm --shell /bin/bash --create-home

Then, setup the environment in the new user's home directory /srv/scm. If you are going to work as www-data, create the folder on your own switch to that user and skip the chmod commands below. You also could spare the additional subfolder finroc.org. However, it comes in handy if you want to create a parallel environment for e.g. testing.

We provide you with the basic environment used on finroc.org as mercurial repository https://finroc.org/hg/admin_finroc_server which itself requires some scripts in a checked out working copy of https://finroc.org/hg/finroc.

~# su - scm_admin
~$ hg clone https://finroc.org/hg/admin_finroc_server finroc.org
~$ cd finroc.org
~/finroc.org$ hg clone -U https://finroc.org/hg/finroc
~/finroc.org$ chmod g+w .
~/finroc.org$ chmod -R g+w finroc

Trac

If you want to use Trac for repository management here are the steps to initialize its environment. Otherwise skip this part.

~/finroc.org$ trac-admin trac initenv
~/finroc.org$ trac-admin trac deploy trac/static
~/finroc.org$ chmod g+w trac/conf trac/db trac/log trac/files/attachments/*
~/finroc.org$ chmod g+rw trac/conf/trac.ini trac/db/trac.db

Configuration

The main configuration must be done in the local etc folder of your scm environment. There is a components.pl which is used for deployment for finroc_get and co. Additionally, two symlinks can be found:

  • hgweb.conf which makes your repositories available via apache
  • trac.ini for Trac if its environment was initialized. Otherwise that link points to nowhere.

First of all, you must replace the following uppercase placeholder in these files, according to your situation:

     YOURDOMAIN -> finroc.org
     SCM_HOME -> /srv/scm/finroc.org

components.pl

Then, you might want to further adapt components.pl. It is basically a file to be included in a perl script as-is and defining a hash that the including script will use as configuration. Here is a short explanation of the keys:

scm
A hash that contains the supported source code management systems. Each itself points to a hash consisting of subdir and base_url.
subdir
The subfolder for all repositories of that scm type.
base_url
The beginning of the URL that can be used to access a repository of that scm type.

scm_precedence
An ordered list that resolves ambiguities if a repository exists with mutliple scm types.
repositories
The absolute path to the manages repositories. For a repository of a specific scm type, its subdir and then its name will be appended to form its path on the server.
output
The absolute path to a folder that will contain the generated files.
distributions
This is itself a hash of hashes that describes the distributions to generate. In the example you see the configuration for development and our first stable release 13.10. The entries use the keys dependency_sources and categories.
dependency_sources
A list of sources as in Finroc's sources.list. The published components in the current distribution can depend on the components that are available through these sources. Dependency information will be published and can be used by finroc_get.
categories
A group of components in the specific distribution for more distinctive selection. Typically at least a category with the name main is available. On finroc.org these categories are mainly formed by distinguishing between licenses, but you can either include or exclude components that match one of a set of patterns. So, the entries of the include and exclude list have an or relation while the regular expressions in the pattern hash have an and relation. A component that matches an exclude pattern will be excluded even if it also matches an include pattern. A pattern may consists of regular expressions on the following component properties:
  • name
  • branch
  • access
  • license
  • scm

Apache Webserver

The next step is to configure Apache to provide access to the repositories. The following snippet stems from /etc/init/apache/sites-available/finroc.org-ssl and shows how mercurial and subversion repositories are made available on finroc.org using wsgi and dav_svn. The additional Allow from rule for /hg restricts access to mercurial clients as webbrowsers (humans) should use the Trac interface in this case. Authentication is realized using macros and must be adapted to your situation. See the apache documentation for more details. The htpasswd file in this configuration is managed by Trac's Account Manager while the svn.authz file is managed by the Repository Manager Plugin for Trac. Change these if you want to do things different.

<VirtualHost *:443>
        ServerName     finroc.org
        ServerAdmin    webmaster@finroc.org

        DocumentRoot   /var/www/finroc.org

        ScriptAlias    /hg     /srv/scm/finroc.org/hgweb/hgweb.cgi

        <Location />
                Options         FollowSymLinks MultiViews
                Order           allow,deny
                Allow from      all
        </Location>

        <Location /hg>
                SetEnvIf        User-Agent mercurial ComeIn=1
                Order           allow,deny
                Allow from      env=ComeIn

                Use             AuthViaFile_or_anonymous "Finroc.org Mercurial Repository - Use anonymous & email address for guest entry" /srv/scm/finroc.org/htpasswd
                Require         valid-user
        </Location>

        <Location /svn>
                DAV svn
                SVNParentPath           /srv/scm/finroc.org/repositories/svn

                Use                     AuthViaFile_or_anonymous "Finroc.org Subversion Repository - Use anonymous & email address for guest entry" /srv/scm/finroc.org/htpasswd
                Require                 valid-user

                AuthzSVNAccessFile      /srv/scm/finroc.org/generated/svn.authz
        </Location>
</VirtualHost>

The following snippets from /etc/init/apache/sites-available/finroc.org show how the component lists are made available as well as Trac.

<VirtualHost *:80>
        ServerName      finroc.org
        ServerAdmin     webmaster@finroc.org

        DocumentRoot    /var/www/finroc.org

        <Location />
                Options         FollowSymLinks Multiviews
                Order           allow,deny
                Allow from      all
                RedirectMath    ^/$     http://www.finroc.org
        </Location>
</VirtualHost>
<VirtualHost *:80>
        ServerName      www.finroc.org
        ServerAdmin     webmaster@finforc.org

        DocumentRoot    /var/www/finroc.org

        Alias           /chrome/site    /srv/scm/finroc.org/trac/site
        Alias           /chrome         /srv/scm/finroc.org/trac/static/htdocs
        WSGIScriptAlias /               /srv/scm/finroc.org/trac/static/cgi-bin/trac.wsgi

        <Location />
                SetEnv          trac.locale     de_DE.UTF-8
                Options         FollowSymLinks MultiViews
                Order           allow,deny
                Allow from      all
        </Location>
</VirtualHost>

The component lists are symlinks in /var/www/finroc.org:

~# ls -l /var/www/finroc.org
lrwxrwxrwx 1 root root   35 Oct 24 17:29 13.10 -> /srv/scm/finroc.org/generated/13.10
lrwxrwxrwx 1 root root   41 Oct 25 18:12 development -> /srv/scm/finroc.org/generated/development

Trac

After configuring Trac to your needs following its documentation, the Repository-Manager-Plugin must be enabled. In our configuration it will create all repositories in /srv/scm/finroc.org/repositories, the repository owner will not automatically be a maintainer and users may fork repositories only once as a private clone for themselves. Therefore, edit /srv/scm/finroc.org/etc/trac.ini so it contains the following lines:

[components]
repo_mgr.* = enabled
tracext.hg.* = enabled
tracopt.versioncontrol.svn.* = enabled

[repository-manager]
base_dir = /srv/scm/finroc.org/repositories
owner_as_maintainer = false
restrict_dir = true
restrict_forks = true
svn_authz_file = /srv/scm/finroc.org/generated/svn.authz
svn_post_commit = /srv/scm/finroc.org/hooks/svn/post-commit
svn_pre_commit = /srv/scm/finroc.org/hooks/svn/pre-commit

[ticket]
general_workflow = ConfigurableTicketWorkflow
workflow = PullRequestWorkflowProxy

[trac]
authz_file = /srv/scm/finroc.org/generated/source.authz
authz_module_name =
permission_policies = AuthzSourcePolicy, DefaultPermissionPolicy, LegacyAttachmentPolicy

To make this setup work remove some default permissions from the anonymous user as they will be granted by the AuthzSourcePolicy.

~/finroc.org$ trac-admin trac
> permission remove anonymous BROWSER_VIEW
> permission remove anonymous CHANGESET_VIEW
> permission remove anonymous FILE_VIEW
> permission remove anonymous LOG_VIEW

Now grant REPOSITORY_ADMIN to a responsible user and let him go to http://YOURDOMAIN/repository/create and create or clone repositories that should be hosted on your server. After editing etc/sources.list in your Finroc working copy, you should be able to install you own components.

As you are now using the RepositoryManagerPlugin that introduced maintainer, forks and pullrequests you also should uncomment the pullrequest hook and allow_push lines in etc/hgweb.conf.

Cronjobs

cron can be used to perform some regular actions. One is to clean the generated components lists from renamed or deleted entries. Run crontab -e (in this example as user scm_admin) and add the following line:

0   4   *   *   *    /srv/scm/finroc.org/bin/rewrite_component_lists > /dev/null
Last modified 3 years ago Last modified on 19.07.2015 17:21:25