sge

class anadama2.grid.sge.SGE(partition, tmpdir, benchmark_on=None, options=None, environment=None)[source]

This class enables the Workflow class to dispatch tasks to Sun Grid Engine and its lookalikes. Use it like so:

from anadama2 import Workflow
from anadama2.sge import SGE

ctx = Workflow(grid_powerup=SGE(queue="general"))
ctx.do("wget "
       "ftp://public-ftp.hmpdacc.org/"
       "HMMCP/finalData/hmp1.v35.hq.otu.counts.bz2 "
       "-O @{input/hmp1.v35.hq.otu.counts.bz2}")

# run on sge with 200 MB of memory, 4 cores, and 60 minutes
t1 = ctx.grid_do("pbzip2 -d -p 4 < #{input/hmp1.v35.hq.otu.counts.bz2} "
                 "> @{input/hmp1.v35.hq.otu.counts}",
                 mem=200, cores=4, time=60)

# run on sge on the serial_requeue queue
ctx.grid_add_task("some_huge_analysis {depends[0]} {targets[0]}",
                  depends=t1.targets, targets="output.txt",
                  mem=4000, cores=1, time=300, partition="serial_requeue")


ctx.go()
Parameters:
  • partition (str) – The name of the SLURM partition to submit tasks to
  • tmpdir (str) – A directory to store temporary files in. All machines in the cluster must be able to read the contents of this directory; uses anadama2.picklerunner to create self-contained scripts to run individual tasks and calls srun to run the script on the cluster.
  • benchmark_on – Option to turn on/off benchmarking

: type benchmark_on: bool

Parameters:options (str) – Grid specific options to apply to each job