Module orngServerFiles
allows users to download files from a common
repository residing on the Orange server. It was designed to simplify the download and updates of
external data sources for Orange Genomics add-on. Furthermore, an authenticated
user can also manage the repository files with this module.
Orange server file repository was created with the intend to store larger files that do not come with Orange installation, but
may be required from the user when running specific Orange functions. A typical example is Orange bioinformatics package,
which relies on large data files storing genome information. These do not come pre-installed, but are rather downloaded
from the server when needed and stored in the local repository. Module orngServerFiles provides low-level functionality
to manage these files, and is used by Orange modules to locate the files from the local repository and update/download them when
and if needed.
Each file managed by orngServerFiles
is described by domain and
the file name. Domains are like directories - a place where files are put in.
Local file management
Downloaded files are saved under Orange's settings directory, subdirectory
buffer/bigfiles
. For each new domain a subdirectory containing
the downloaded files is created. With every download a corresponding
info file is created with bearing the same name but added an extension plus ".info".
Info fail contains file's title, tags, size and date and time of the upload.
Methods used for managing local files
- download(domain, filename, serverfiles=None, callback=None, extract=True, verbose=True)
- Downloads file from the repository to local orange installation. To download
files as an authenticated user you should also pass an instance of
ServerFiles
class.
Callback can be a function without arguments. It will be called once for each downloaded percent of file: 100 times for the whole file.
- listdomains()
- List all file domains in the local repository.
- listfiles(domain)
- List all files from a domain in a local repository.
- info(domain, filename)
- Returns a dictionary containing downloaded file information. Keys: title, tags, size, datetime.
- allinfo(domain)
- Goes through all files in a domain on a local repository and returns a dictionary, where keys are names of the files and values are their information.
- search(substrings, caseSensitive=False, inTag=True, inTitle=True, inName=True)
- Function searches for files in the local repository where all substrings in a list are contained
in at least one chosen field (tag, title, name). It returns a list of tuples: first tuple element
is the domain of the file, second its name.
- localpath(domain [,filename])
- Returns a path for the domain in the local repository. If argument
filename
is given, returns a path to corresponding file.
- localpath_download(domain, filename, **kwargs)
- Returns local path for the given domain and file. If file does not exist on the local machine, download it. Additional arguments are passed to the
download
function.
- remove_domain(domain [,force])
- Removes a domain. If force is True, domain is removed even
if it is not empty (contains files).
- remove(domain, filename)
- Removes a file from repository.
- update(domain, filename, access_code=None, verbose=True)
- Downloads the corresponding file from the server and places it in the local repository, but only if the server copy of the file is newer or the local copy does not exist. An optional
access_code
can be given to allow server access if the file is protected.
ServerFiles: the repository
To work with the repository, you need to create an instance of
ServerFiles
object. To access the repository as an authenticated user,
a username and password should be passed to the constructor.
All password
protected operations and transfers are secured by SSL; this secures both password and content.
Repository files are set as protected when first uploaded:
only authenticated users can see them. They need to be unprotected
for public use.
- ServerFiles(username=None, password=None, access_code=None)
- Creates a ServerFiles instance. Pass your username and password to use
the repository as an authenticated user. If you want to use your access code
(as an non-authenticated user), pass it also.
Non-authenticated users
- download(domain, filename, target, callback=None)
- Downloads file from the repository to a given target name. Callback can be a function without arguments. It will be called once for each downloaded percent of file: 100 times for the whole file.
- listdomains()
- List all domains on repository.
- listfiles(domain)
- List all files in a repository domain.
- info(domain, filename)
- Returns a dictionary containing repository file info. Keys: title, tags, size, datetime.
- allinfo(domain)
- Goes through all accessible files in a given domain and returns a dictionary, where key
is file's name and value its info.
- search(substrings, caseSensitive=False, inTag=True, inTitle=True, inName=True)
- Function searches for files on the repository where all substrings in a list are contained
in at least one choosen field (tag, title, name). It returns a list of tuples: first tuple element
is the file's domain, second its name. As for now the search is performed locally, therefore
information on files in repository is transfered on first call of this function.
- downloadFH(domain, filename)
- Returns a file handle to the file that we would like to download.
Authenticated users
Authenticated users can use methods described in "Non-authenticated users"
section, but they can also use the functions that access and modify protected files.
- create_domain(domain)
- Creates a repository domain.
- remove_domain(domain [,force])
- Removes a domain. If force is True, domain is removed even
if it is not empty (contains files).
- remove(domain, filename)
- Removes a file from the repository.
- protection(domain, filename)
- Returns file protection. Legend: "0" - public use, "1" - for authenticated
users only, anything else represents a specific access code.
- protect(domain, filename [, access_code])
- Hide file from non-authenticated users. If an access code (string) is passed, the file
will be available to authenticated users and non-authenticated users with that access
code.
- unprotect(domain, filename)
- Put a file into public use.
- upload(domain, filename, localfile [, title, tags])
- Uploads a file "localfile" to the domain where it is saved with
filename "filename". If file does not exist yet, set it as
protected. Parameter localfile can be a file handle open for reading or
a file name.
Examples
Downloading and listing files
Listing local files, files from the repository and downloading all
available files from domain "demo".
import orngServerFiles
repository = orngServerFiles.ServerFiles()
print "My files", orngServerFiles.listfiles('demo')
print "Repository files", repository.listfiles('demo')
print "Downloading all files in domain 'test'"
for file in repository.listfiles('demo'):
print "Datetime for", file, repository.info('demo', file)["datetime"]
orngServerFiles.download('demo', file)
print "My files after download", orngServerFiles.listfiles('demo')
print "My domains", orngServerFiles.listdomains()
Output for first run (and current repository state):
My files []
Repository files ['orngServerFiles.py', 'urllib2_file.py']
Downloading all files in domain 'test'
Datetime for orngServerFiles.py 2008-08-20 12:25:54.624000
Datetime for urllib2_file.py 2008-08-20 12:25:54.827000
My files after download ['urllib2_file.py', 'orngServerFiles.py']
My domains ['demo']
Creating a domain, uploading files
A similar domain as "demo" in previous example can be built as follows.
import orngServerFiles
ordinary = orngServerFiles.ServerFiles()
authenticated = orngServerFiles.ServerFiles(username, password)
try:
authenticated.remove_domain('demo2', force=True)
except:
pass
authenticated.create_domain('demo2')
authenticated.upload('demo2', 'orngServerFiles.py', 'orngServerFiles.py', \
title="orngServerFiles client", tags=["basic", "fileManagement"])
authenticated.upload('demo2', 'urllib2_file.py', 'urllib2_file.py')
print "Non-authenticated users see:", ordinary.listfiles('demo2')
print "Authenticated users see:", authenticated.listfiles('demo2')
authenticated.unprotect('demo2', 'orngServerFiles.py')
authenticated.unprotect('demo2', 'urllib2_file.py')
print "Non-authenticated users now see:", ordinary.listfiles('demo2')
print "orngServerFiles.py file info:"
import pprint; pprint.pprint(ordinary.info('demo2', 'orngServerFiles.py'))
Output:
Ordinary users see: ['']
Authenticated users see: ['orngServerFiles.py', 'urllib2_file.py']
Non-authenticated users now see: ['orngServerFiles.py', 'urllib2_file.py']
orngServerFiles.py file info:
{'datetime': '2008-08-26 10:30:05.373000',
'size': '10165',
'tags': ['basic', 'fileManagement'],
'title': 'orngServerFiles client'}