[Simfactory] Presubmitting

[Erik Schnetter]

On Jun 30, 2009, at 15:21:40, Ian Hinder wrote:

> Hi,
>
> I currently create and submit simulations with "create-submit".  If I
> want to submit a chain of simulations, I then use "submit --presubmit"
> for each one.  If I want to add to the chain, I then either have to do
> "submit --presubmit" again if there is at least one of the jobs in the
> queue, or "cleanup-submit" if they have all finished.
>
> 1. Would it be possible to specify the number of "presubmits" at the
> "create-submit" stage?  And also when using the "submit --presubmit"
> command?

Would a --count argument be good for this?  Or should the --presubmit  
option take a numeric argument?  If so, should that argument be  
optional?

> 2. It would be nice to be able to "extend" the chain of simulations in
> a uniform way which did not depend on whether the jobs had finished
> yet or not.

You mean something like maybe-cleanup-submit-presubmit?

> 3. Since each of these commands (when run remotely) causes an ssh
> connection, it can take quite a while.  It would be nice to be able to
> "continue" more than one simulation in a single command.  This would
> be helpful when managing a batch of simulations.

Here is a bit of brain storming:

sim remote kraken { submit A; submit B; submit C; }
sim remote kraken submit { A B C }
sim remote kraken { cleanup submit } A
sim remote { kraken ranger } { cleanup submit } { A B C }

sim remote kraken foreach S (A B C) cleanup-submit S

Actually, the following works already:

./simfactory/sim remote damiana execute 'for id in 0006 0007 0008;  
do ./simfactory/sim cleanup $id; done'

I think that the simulation factory should not try to duplicate what  
the shell can already do.  I don't want to invent a new language -- it  
would be better to make the simulation factory work smoothly with the  
shell, as the shell has already a for command.  Alternatively, the  
simfactory commands could be exported as perl (or python) commands,  
and you could then use perl or python looping constructs.

> 4. Whenever I run create-submit, I invariably want to perform a sync
> beforehand to make sure that the parfiles directory on the remote
> machine is up to date.  However, only the parfiles directory needs to
> be synced, not the whole tree.  Would it be possible for this to be
> implemented in simfactory?

What about a new command sync-parfiles?  Or maybe sync -- 
subdir=parfiles?

> 5. When I run "sim stop", only the first job in a chain is stopped.
> Is there a way to make it stop all the jobs?  Maybe a different
> command, or an option to "stop"?


The stop command aborts a job, removing it from the queue.  What  
should happen to the other jobs?  Should they be held or deleted?   
(Maybe either, depending on an option.)  What should be the default?   
(They should probably be removed from the queue.)  Should there also  
be "hold" and a "release" commands for jobs?

-erik

-- 
Erik Schnetter <schnetter at cct.lsu.edu>   http://www.cct.lsu.edu/~eschnett/

My email is as private as my paper mail.  I therefore support encrypting
and signing email messages.  Get my PGP key from <http://pgp.mit.edu>.



-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 203 bytes
Desc: This is a digitally signed message part
Url : https://mail.cct.lsu.edu/mailman/private/simfactory/attachments/20090707/ca6c1514/attachment.bin