Fetching scripts

Overview

I'm considering building in a mechanism for automatically fetching scripts from PyMOLWiki. The goal is to allow users to say

fetch findSurfaceResidues, type=script
findSurfaceResidues doShow=True, cutoff=0.5

The convenience benefits are obvious, especially for new users, and I think that lowering the barrier to script usage will greatly increase both the number of people who use various scripts and the incentive to place scripts on the wiki (especially if the fetch mechanism makes it easy for script authors to provide a citation/DOI/etc.).

Issues

Security

Running untrusted code is trouble. Some ideas

• MediaWiki allows us to protect pages so that only administrators can edit them. We could protect all approved scripts.
• Alternately, we could have a protected page that links to the approved scripts.
• I lean towards using both of those options. Just requiring the page to be protected could cause trouble if there's an unrelated page that happens to both be protected and have a security leak due to the fact that it was never intended to be a fetchable script.
• Maybe a hybrid system where scripts have a development version and a release version? I don't want to make too much overhead, though. Current idea: setting secure=0 means you get the development version.
• We should print a warning each time a new script is fetched anyway
• Can fetched scripts persist across saved sessions? It's easy enough to save them in fetch_path.
• Plugins? This is probably more worth considering for a future version, but it would be nice to be able to load plugins as well. Since plugins are (now) installed permanently, we have to think carefully about the implications.

Convenience options

The main benefit is to make things as convenient and easy as possible, especially for new users.

• Local cache. This would make reloading scripts with each new session easier and faster. You could then stick a bunch of "fetch" lines in your pymolrc. fetch_path.
• A command to list all available scripts?

Validation

• How will users know that their script is doing the correct thing
• Perhaps we should have two classes of scripts: approved and validated

Format

My guess is that we'll require fetchable scripts to follow a certain format on the wiki pages. That should include some metadata like

• Version number. This makes debugging easier and it makes smart caching possible
• Citation. Script authors should be able to provide a preferred citation, DOI, etc. One of the benefits is to get script authors more credit.
• Documentation. Or should this be handled in the doc string?

I think all of these things should be function attributes

def myScript(a,b,c):
    """ Documentation for myScript goes here """
    myScript.citation = "I. Coder, A Journal, 2011, Vol. 2, Issue 1, pages 64-66"
    myScript.version = 1.2

    blah
    blah

cmd.extend('myScript',myScript)

Implementation

• This will obviously be written in Python.
• We'll probably make use of some screen scraping library. I don't know the state of the art here, but I've seen at least the following, and would love some comments:
  • Generic interfaces
    • scrapy (looks reasonable, this page may also be useful.
    • Beautiful Soup
    • lxml
    • mechanize
    • webscraping
    • urllib(2?) with regular expressions for that old-school feel?
  • Specific interfaces
    • PyWikipedia
    • python-wikitools (this looks good to me!) can never be bundled with PyMOL due to GPL v3.
    • Other pages listed here