Generate p2 composite repository metadata from Jenkins (Hudson) jobs on the fly

Serving p2 build artifacts from Jenkins is complicated if builds from multiple Jenkins jobs should be aggregated into a master p2 composite repository. A rather simple approach to this problem is listed below, that uses Apache to generate compositeArtifacts.xml and compositeContent.xml on the fly.
We (vogella) uses this to generate repos for our various Jenkins jobs (e.g the content of compositeArtifacts.xml is generated).
  • Install Apache and point DocumentRoot to Jenkin’s jobs/ directory
    • Enable cgi scripts in Apache
      • On Debian or Ubuntu this is `a2enmod cgid`
    • AllowOverride all for the jobs/ directory (needed in order for .htaccess to work)
  • Clone https://github.com/lemmy/p2CompositeRepoGeneratorinto jobs/ and fix file ownership/permissions
    • Cloning into a non-empty directory isn’t allowed, thus move the content of p2CompositeRepoGenerator/ into jobs/ (including .git)
    • `chmod +x composite*.xml`
    • `chown jenkins:nogroup composite*.xml .htaccess p2.index`
  • Symlinking composite*.xml from individual jobs generates a p2 composite repository for all builds of that Jenkins jobs (contrary to all builds of all jobs)
  • Open a browser on the corresponding DocumentRoot url and check the content of compositeArtifacts.xml and compositeContent.xml
    • If no repositories show up, check that Jenkins even archives non-zipped p2 repository artifacts during the post build action
    • An internal server errors can best be diagnosed by opening the Apache error log
  • The final directory structure should look similar to (the second part lists the structure for our I-build of the eclipse.platform.releng.aggregator) :
jenkins@build:~$ ls -lA jobs/
total 72
drwxr-xr-x  8 jenkins nogroup 4096 Mar 27 10:18 .git
-rw-r--r--  1 jenkins nogroup   28 Mar 27 09:06 .gitignore
-rw-r--r--  1 jenkins nogroup  402 Mar 27 09:02 .htaccess
drwxr-xr-x  5 jenkins nogroup 4096 Mar 27 10:42 C-MASTER-e4tools-0.14
drwxr-xr-x  5 jenkins nogroup 4096 Mar 27 10:42 I-MASTER-eclipse.platform.releng.aggregator
-rwxr-xr-x  1 jenkins nogroup 3192 Mar 27 10:18 compositeArtifacts.xml
lrwxrwxrwx  1 jenkins nogroup   22 Mar 27 09:02 compositeContent.xml
	-> compositeArtifacts.xml
-rw-r--r--  1 jenkins nogroup  129 Mar 27 09:02 p2.index

jenkins@build:~$ ls -lA jobs/I-MASTER-eclipse.platform.releng.aggregator/
total 68
drwxr-xr-x   5 jenkins nogroup  4096 Mar 27 10:32 builds
lrwxrwxrwx   1 jenkins nogroup    44 Mar 26 13:14 compositeArtifacts.xml
	-> /var/lib/jenkins/jobs/compositeArtifacts.xml
lrwxrwxrwx   1 jenkins nogroup    42 Mar 26 13:16 compositeContent.xml
	-> /var/lib/jenkins/jobs/compositeContent.xml
-rw-r--r--   1 jenkins nogroup  6458 Mar 26 11:57 config.xml
-rw-r--r--   1 jenkins nogroup     3 Mar 26 11:57 nextBuildNumber
lrwxrwxrwx   1 jenkins nogroup    11 Mar 27 10:42 p2.index -> ../p2.index

Disclaimer: This is probably not the best idea for busy p2 repositories. The script a) scans the directory tree for every invocation and b) sets the timestamp to $now (messing up caches). b3aggr is a much better tool for frequently used repositories.
This entry was posted in Other. Bookmark the permalink.