Building a pypi package index with pypicache-0.2

An index that was built following this example is currently hosted here.

Copy the pypicache sources and some .egg files up to the server that will host your index. If you dont think setuptools is available on the server then simply include a setuptools .egg in the .eggs you place in the index directory.:

scp pypicache-0.2.tar.gz setuptools-0.6c3-py2.4.egg *.egg user@aldis:/srv/pypi-cached/
ssh -l user@aldis
cd /srv/pypi-cached
tar -zxf pypicache-0.2.tar.gz

Do a dry run to see what files will be generated and to test the installation:

user@aldis:/srv/pypi-cached# python pypicache-0.2/indexpackages.py --enable-htaccess -T "cheesboard.wiretooth.com" -U http://svn.wiretooth.com/pypi/ -N .

You will see output that starts something like this:

downloads-server-url: "http://svn.wiretooth.com/"
downloads-path: "/pypi/"
index-url: "http://svn.wiretooth.com/pypi/"
archive formats:gztar, ztar, bztar, zip, tar
Discarding ambiguous project/version interpretation: Pyrex/Tests-0.9.5.1
        Becuase the version part starts with a non digit
Discarding ambiguous project/version interpretation: python/cjson-1.0.2
        Becuase the version part starts with a non digit
Discarding ambiguous project/version interpretation: python/cjson-1.0.3
        Becuase the version part starts with a non digit
Pyrex: 0.9.5.1a [Pyrex-0.9.5.1a.tar.gz]
dry-wrote:/srv/pypi-cached/pyrex-0.9.5.1a.html
Pyrex-Tests: 0.9.5.1 [Pyrex-Tests-0.9.5.1.tar.gz]
dry-wrote:/srv/pypi-cached/pyrex-tests-0.9.5.1.html
SQLAlchemy: 0.3.7dev-r2498, 0.1.7 [SQLAlchemy-0.3.7dev_r2498-py2.4.egg, SQLAlchemy-0.1.7-py2.4.egg]
dry-wrote:/srv/pypi-cached/sqlalchemy-0.3.7dev-r2498.html
dry-wrote:/srv/pypi-cached/sqlalchemy-0.1.7.html
commandplate: 0.1dev1 [commandplate-0.1dev1-py2.5.egg, commandplate-0.1dev1.tar.gz]
dry-wrote:/srv/pypi-cached/commandplate-0.1dev1.html
docutils: 0.5dev, 0.4, snapshot [docutils-0.5dev.tar.gz, docutils-0.4.tar.gz, docutils-snapshot.tgz]

We set the content for master index page <title> using -T. The url of the index was set with -U. This is the url that people should pass using the -i option provided by easy_install when they want to install packages from your index.

Notice the messages regarding the package named Pyrex-Tests. See ticket#12 for details on the relatively simple heuristic that is used to resolve ambiguities.

Assuming the run was successful the output will conclude by listing the apache configuration files that would be written:

dry-wrote:/srv/pypi-cached/simplejson-1.7.1.html
dry-wrote:/srv/pypi-cached/simplejson-1.4.html
dry-wrote:/srv/pypi-cached/simplejson-1.3.html
zc.recipe.egg: 1.0.0b6 [zc.recipe.egg-1.0.0b6-py2.5.egg]
dry-wrote:/srv/pypi-cached/zc.recipe.egg-1.0.0b6.html
zope.interface: 3.2.0.2 [zope.interface-3.2.0.2-py2.4-linux-i686.egg]
dry-wrote:/srv/pypi-cached/zope.interface-3.2.0.2.html
dry-wrote:/srv/pypi-cached/index.html
dry-wrote:/srv/pypi-cached/project-index.txt
dry-wrote:/srv/pypi-cached/apache2-vhost.directives.conf
dry-wrote:/srv/pypi-cached/apache2-dir.directives.conf
dry-wrote:/srv/pypi-cached/.htaccess

The dry-wrote prefix on all the generated file names simply confirms that -N, --dry-run is in effect and the named file was not written to disc. If you want to pre view the config files, without actualy writing to disc, include the options -N --log-config=INFO. Similarly, to preview the content of the index pages, you can use --log-content=INFO

Note that if -T name is a '.' separated prefix of of -U index-url then you get a ServerAlias for it in the generated apache vhost configuration. This is not always what you want and the generate ServerAlias feature may be removed in a future release.

Run it for real:

user@aldis:/srv/pypi-cached# python pypicache-0.2/indexpackages.py --enable-htaccess -T "cheesboard.wiretooth.com" -U http://svn.wiretooth.com/pypi/ .
downloads-server-url: "http://svn.wiretooth.com/"
downloads-path: "/pypi/"
index-url: "http://svn.wiretooth.com/pypi/"
archive formats:gztar, ztar, bztar, zip, tar
Discarding ambiguous project/version interpretation: Pyrex/Tests-0.9.5.1
        Becuase the version part starts with a non digit
Discarding ambiguous project/version interpretation: python/cjson-1.0.2
        Becuase the version part starts with a non digit
Discarding ambiguous project/version interpretation: python/cjson-1.0.3
        Becuase the version part starts with a non digit
Pyrex: 0.9.5.1a [Pyrex-0.9.5.1a.tar.gz]
wrote:/srv/pypi-cached/pyrex-0.9.5.1a.html
Pyrex-Tests: 0.9.5.1 [Pyrex-Tests-0.9.5.1.tar.gz]
wrote:/srv/pypi-cached/pyrex-tests-0.9.5.1.html
SQLAlchemy: 0.3.7dev-r2498, 0.1.7 [SQLAlchemy-0.3.7dev_r2498-py2.4.egg, SQLAlchemy-0.1.7-py2.4.egg]
wrote:/srv/pypi-cached/sqlalchemy-0.3.7dev-r2498.html
wrote:/srv/pypi-cached/sqlalchemy-0.1.7.html
<snip>

And this time we see wrote instead of dry-wrote

Setup a vhost container, or update an existing one, to pull in the generated configs. Assuming you have a debian/ubuntu style apache configuration you can use something like this.:

echo "
<VirtualHost *>
        ServerAdmin webmaster@localhost
        <Directory />
                Options None
                AllowOverride None
        </Directory>
        Include /srv/pypi-cached/apache2-vhost.directives.conf
        RewriteLogLevel 3

        # If you need to host this under a different document root, do
        # something like this:
        # DocumentRoot /somewhere/else
        # Alias /pypi /somewhere/else

        Include /srv/pypi-cached/apache2-dir.directives.conf

        ErrorLog /var/log/apache2/error.log

        # Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel debug

        CustomLog /var/log/apache2/access.log combined
        ServerSignature On

</VirtualHost>" > /etc/apache2/sites-available/001-cheeseboard.myhost.conf


a2ensite 001-cheeseboard.myhost.conf
/etc/init.d/apache2 reload

If you get Invalid command 'RewriteEngine' errors then you need to enable mod_rewrite AND force a full apache restart.:

a2enmod rewrite; /etc/init.d/apache2 restart

If you don't have server access, or if you dont have control over the server level config you need to sweet talk your administrator into enabling .htaccess for your package cache. Point them at the generated config files (which are well commented) and you might get lucky.