pyrun: sys.path discovery and execution of python modules and packages.

License:MIT
Name:runtools
Version: 0.0.2.dev
Author: Robin Bryce
Author-email:robinbryce@gmail.com
Requires-Python:
 2.5
Copyright: Copyright (c) 2007 Robin Bryce, All rights reserved
Classifiers:License :: OSI Approved :: MIT License
Classifiers:Development Status :: 3 - Alpha
Classifiers:Programming Language :: Python
Classifiers:Intended Audience :: Developers
Classifiers:Topic :: Software Development :: Libraries :: Python Modules
Classifiers:Natural Language :: English

Abstract

A convenient way to run python modules, packages and scripts with a dynamically discovered sys.path. Particularly useful when you want to make use of packages without installing them first, or to circumvent bootstrapping issues with complicated collections of python software.

Contents

pyrun

You can access the most commonly useful features of pyrun, without installing the project, by running the pyrun.py file directly.

For example, the following:

cd ~
wget http://svn.wiretooth.com/svn/open/pyrun-trunk/pyrun.py
``python ~/pyrun.py  ~/my/python/libs ~/my/python/scripts/go.py``

Is all you need in order to run the go.py module with a sys.path automatically discovered from directories under ~/my/python/libs. You can list an arbitrary number of directories and paths to actual python files. The order you list them controls the order in which the path extension entries are built. The resulting path will not contain any duplicates. Each entry in the path extension will be a legitimate import path.

Note:
A setup.py file is provided - with suitable egg entry point declarations - if you prefer this.

For each argument which identifies a python file pyrun will locate the root package directory and add that to the path. The absolute dotted module name of the first python file you identify in this way is the module that will, by default, be executed as __main__. You can explicitly override this choice by using pyrun's -m option.

Note:
root package is the parent directory of the "top most package" see distutils python terms and distutils distutils terms

The delimiter between the pyrun arguments and options and the options for the target module is the first non option argument encountered after the discovery paths. If that option is a pyrun option (see pyrun --help for the list) then pyrun takes it and passes all remaining arguments to the target module in a suitably massaged sys.argv.

If the target module takes arguments but does not naturally accept an option as its first argument (python setup.py install is the classic example) then you can artificially terminate the pyrun options with --.

For example it is possible to run the setup script of the pyrun project in the following ways

  • If you have setup tools installed:

    python setup.py bdist_egg
    
  • If your python distribution has not removed distutls from the python standard library:

    python setup.py sdist
    
  • If you have a copy of the setuptools egg in ../python/eggs:

    python pyrun.py ../python/eggs -m setup bdist_egg
    python pyrun.py ../python/eggs setup.py -- bdist_egg
    

If you have a directory which contains a docutils source tree or installation then adding that to the discovery path will let the setup.py script build this documentation.

If none of the non option arguments identify a python module file and you dont explicitly select one using -m then pyrun will simply print the path it has discovered and exit. You can force just print the path using -p or -P

pyrun is reasonably smart in respect of python egg distributions. When multiple egg distributions of the same project are found on the discovery path only the best version found is included in the path extension. Eggs which are not compatible with the current python interpreter are ignored. The measure of best egg for a project uses the same algorithm as used by the pkg_resources.py module distributed by the setuptools project.

Note:
The current version does not filter out incompatible platforms for eggs that contain c extensions - see pyrun.filter_best_eggs if you have time on your hands, its not to much work to add this check.

The issue tracker for this package can be found at:

http://trac.wiretooth.com/public/wiki/pyrun

When opening a ticket please assign it to the pyrun component or, at least, mention pyrun in your ticket summary.

Installation

Installing this package using python setup.py install will generate some convenience scripts. You can access the most commonly useful features of pyrun, without installing the package, by running the pyrun.py file directly.

Command line interface

Usage: pyrun.py [-nidDpP] [BASEPATH(s)][-m mod.name | '--'] [TARGET-OPTIONS]

In most cases the solo '--' is not required. It tends to be useful when you implicitly select the module to run AND you want to pass a non option argument as the first value in the command line for that module. It can also be necessary when the target module has short options, without long-name alternatives, which collide with those defined for pyrun.

Discover python packages and modules under PATH. Run the first module file named in PATH OR explicitly nominated using the -m option.

NOTE: Any option that is marked [NYI] is Not Yet Implemented.

Options:
-h, --help show this help message and exit
--log-level=LEVEL
 [default:WARNING] set the logging level, any string which names a log level which is defined by the logging package is allowed. For example any of CRITICAL, WARNING, INFO and DEBUG (in increasing order of verbosity)
-q Suppress all warnings about missing paths etc. Useful when you are using speculative paths and are using -p or -P to print the discoverd path.
-p Print the discovered path
-P Print the discovered path in a PYTHONPATH compatible format
-n NORUN. Don't run any of the modules implied by module file references in the discovery path.
-C SCRIPT Identify a python SCRIPT to execute. The script need not have file extension but it must contain leagal python code. This option trumps -m. This option should only be necessary when the launcher for the python program you wish to run contains significant functionality. No additions are made to the discovery path or sys.path as a result of using this option. If the target script imports a related package you will need to include additional non option arguments to discover its path.
-m MODULE Explicitly select a module to run. (trumped by -S)
-d DEBUG session. Use pdb.runeval on the module code in order to enter an interactive debug session at the first python statement of the target module
-D POSTMORTEM debugging. If the target raises an exception, start a postmortem pdb debugging session.
-i INTERACTIVE session with prepared sys.argv and sys.path.
-c STATEMENT Update sys.argv and sys.path then execute the statement in a new, clean, module context.
-x EXCLUDE Exclude one or more directories, separated by ":", from the discovery path.
-X PRUNE Prune all paths which contain this value from the set of paths which were discovered. Specify multiple -X options if you wish too prune based on more than one string.

pyrun.discover_and_run, pyrun.discover_path

All of the command line programs provided by this package are essentially thin wrappers around the discover_and_run or discover_path functions. This section provides doctest based discussion and examples. These examples are also part of the pyrun test suite.

Lets setup a fake set of python packages and use it to illustrate some basics.

>>> import os, sys
>>> from os.path import join
>>> import pyrun
>>> from _testutils import mktmpfiles, printpaths
>>> sysver_cur = sys.version[:3]
>>> sysver_notcompatible = '.'.join(map(str,
...   [sys.version_info[0], sys.version_info[1] + 1]))
>>> tmpdir, canonicalpaths = mktmpfiles((
...     'A/AA/paaa/__init__.py',
...     'A/AA/paaa/paaaa/',
...     'B/BB/paaa/__init__.py',
...     'B/BB/paaa/paaaa/',
...     'B/BB/pbbb/pbbbb/__init__.py',
...     'B/pc/__init__.py',
...     'Modules/module_a.py',
...     'Modules/module_b.py'),
...
...     prefix='pyrun-'
...     )

This search gives an os dependent ordering of all packages under A and B lexicaly ordered depth first is common. Note that Modules is ignored.

>>> pth, minfos, ia = pyrun.discover_and_run(['pyrun', tmpdir], run_module=False)
>>> printpaths(pth, strip=tmpdir)
A/AA
B
B/BB
B/BB/pbbb

This search forces paths under 'B' to be considered first, again note that Modules is ignored.

>>> pth, minfos, ia = pyrun.discover_and_run(['pyrun',
...     join(tmpdir, 'B'), tmpdir], run_module=False
...     )
>>> printpaths(pth, strip=tmpdir)
B
B/BB
B/BB/pbbb
A/AA

Modules is not present in either of the above resulting paths because python module files are ignored unless they were explicitly mentioned in the search.

>>> pth, minfos, ia = pyrun.discover_and_run(['pyrun', tmpdir,
...     join(tmpdir, 'Modules', 'module_a.py')], run_module=False)
>>> printpaths(pth, strip=tmpdir)
Modules
A/AA
B
B/BB
B/BB/pbbb

This search mentions two module files explicitly, in addition to the root of our fake tree.

>>> pth, minfos, ia = pyrun.discover_and_run(['pyrun', tmpdir,
...     join(tmpdir, 'Modules', 'module_a.py'),
...     join(tmpdir, 'Modules', 'module_b.py')], run_module=False)
>>> printpaths(pth, strip=tmpdir)
Modules
A/AA
B
B/BB
B/BB/pbbb

The discovery algorithm currently forces paths discovered from module files ahead of those discovered for package files, however the paths for module files still follow the order in which the module files are listed in the search path. Note that for module files which are contained in one directory you only need to include one in the search but including many from the same directory does not cause duplicates. More elaborate schemes for addressing the shadowning problem are definitely viable, especially given the module loading facilities in the runpy standard lib module. I suspect the "force to front" rule is essential for covering my use case which is "just fix my path and run that script damit!"

Lets drop some eggs into the mix. We dont use real eggs for these examples because the discovery of eggs does not look at egg contents - only file and directory names.

>>> tmpdir, canonicalpaths = mktmpfiles((
...     'Modules/foo-0.1-py%s.egg' % sysver_cur,
...     'Modules/foo-0.2-py%s.egg' % sysver_cur,
...     'Modules/foo-0.2-py%s.egg-link' % sysver_cur,
...     'Modules/zzz-0.2-py%s.egg-link' % sysver_cur,
...     'Modules/foo-0.3-py%s.egg' % sysver_notcompatible,
...     'A/bar-0.1-py%s.egg' % sysver_cur,
...     'B/bar-0.2-py%s.egg' % sysver_cur),
...     tmpdir=tmpdir
...     )

A full search,

>>> pth, minfos, ia = pyrun.discover_and_run(['pyrun', tmpdir], run_module=False)
>>> printpaths(pth, strip=tmpdir)
A/AA
B/bar-0.2-py2.5.egg
B
B/BB
B/BB/pbbb
Modules/foo-0.2-py2.5.egg

A search that considers 'B' first

>>> pth, minfos, ia = pyrun.discover_and_run(['pyrun',
...     join(tmpdir, 'B'), tmpdir], run_module=False
...     )
>>> printpaths(pth, strip=tmpdir)
B/bar-0.2-py2.5.egg
B
B/BB
B/BB/pbbb
A/AA
Modules/foo-0.2-py2.5.egg

Note that irrespective which order we visit the A and B sub trees, we always list bar-0.2. The search discards duplicate egg names, retaining the first and - irispective of the visit order - always lists the best version of each egg.

Finaly note that foo-0.3 is not listed. This is not a bug - the search ignores eggs whose major & minor revisions dont match the current interpreter. But, for now, no special care is taken to deal with platform specific eggs (linux-i686 vs whatver windows eggs use.)

ChangeLog

0.1.2:
Changed distribution project name to runtools. pyrun remains as a standalone script. The runtools package contains a number of utilities that are commonly usefull in conjunction with pyrun.
0.1.1:
  • Added support for directory exclusions and path prunes.
    • exclusion prevents discovery descending into a directory if that directory startswith an element in the exclusion set (see new option -x).
    • prune operates on the result of the discovery. Any path which contains one of the strings specified by a prune is removed from the discovered path.

    exclusion hides directories from the discovery process, prune removes results after the fact. exclusions are enabled by -x prunes are enabled by -X empty strings are always removed from -x and -X.

    There is precisely one mechanism which enables packages under an exclusion path to be explicitly added back in: All explicitly identified python modules are exempted from the exclusion (but not the prune)

  • Decided to use logging instead of print for notification, I had avoided this previously because I did not want to polute the logging configuration of the target app. I consider this change provisional, if it causes to much trouble I will revert to print

  • Added support for including .pth files in the discovery phase. This makes it possible to insert paths to directories which do not contain an __init__.py or are the dirname() of an explicitly referenced module file.

  • bugfix: propagate source file name (or sensibly invented filename) to the __file__ attribute of the code instance that becomes our __main__

  • -d works with -m and -c, should also work with -s but have not tried yet.

  • -d option uses pdb.runeval rather than set_trace making for considerably simpler target debugging. May consider introducing an "eager" -d variant that behaves like 0.1 (set_trace in pyrun.py) later.

  • Allow the target to run when -d is in effect, without requiring user to manaully do opts.d = False

  • By default suppress -D if -d is in effect.

  • Added -c, its much like python -c

  • The run script option changed from -S to -C exist on the file system. -q suppreses the warnings

0.1d:
  • -S option to explicitly run a python script. uses compiler.compile to load and compile the code from an arbitrary file. uses runpy._run_module_code to execute the resulting module as __main__. NOTE No discovery or sys.path manipulation is performed on the file specified with this option. works for plain old python scripts and executable scripts that lack an extension. . The chief motivation for this is to support python projects that distribute substantive scripts rather than using the setuptools entry_points 'wrapper' script approach. (For example see the 0.2.1 source tarbal for pyflakes).
  • documentation tidy up
0.1c:
  • Use the parent directory of the top package directory for each explicitly listed python file in the discovery path. This means you need to always provide fully qualified module names to -m. This change is necessary to allow the targeted modules imports to work correctly.

0.1b:

  • path discovery and python module execution

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.