[Bigjob-users] [saga-users] Local installation of SAGA and running example of BIGJOB

Kundan Kadam kkadam1 at tigers.lsu.edu
Wed Dec 7 14:44:51 CST 2011 

I changed COORDINATION_URL from "advert://localhost/?dbtype=sqlite3"
to "advert://advert.cct.lsu.edu:8080/". The example makes progress but
gives following error at the end. What should I do?

Start Pilot Job/BigJob at: fork://localhost
DEBUG:root:init BigJob w/: advert://advert.cct.lsu.edu:8080/
DEBUG:root:['/home/kkadam/../', '/home/kkadam',
'/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/BigJob-0.3.2-py2.7.egg',
'/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/redis-2.2.4-py2.7.egg',
'/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/virtualenv-1.6.4-py2.7.egg',
'/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/threadpool-1.2.7-py2.7.egg',
'/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/uuid-1.30-py2.7.egg',
'/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/redis-2.2.4-py2.7.egg',
'/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/threadpool-1.2.7-py2.7.egg',
'/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/uuid-1.30-py2.7.egg',
'/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg',
'/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages',
'/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/2.7/site-packages',
'/home/kkadam',
'/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python27.zip',
'/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python2.7',
'/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python2.7/plat-linux2',
'/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python2.7/lib-tk',
'/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python2.7/lib-old',
'/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python2.7/lib-dynload',
'/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python2.7/site-packages',
'/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/2.7/site-packages',
'/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/2.7/site-packages',
'/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/BigJob-0.3.2-py2.7.egg/bigjob']
DEBUG:root:Utilizing ADVERT Backend
DEBUG:root:Parsing URL: advert://advert.cct.lsu.edu:8080/
DEBUG:root:Server: advert.cct.lsu.edu Port 8080 server_connect_url: None
DEBUG:root:initialized BigJob: bigjob:4d1ed72a-2111-11e1-bae6-a4badb0c3696
DEBUG:root:create pilot job entry on backend server:
bigjob:4d1ed72a-2111-11e1-bae6-a4badb0c3696:localhost
DEBUG:root:create advert entry:
advert://advert.cct.lsu.edu:8080/bigjob/4d1ed72a-2111-11e1-bae6-a4badb0c3696/localhost
DEBUG:root:update state of pilot job to: Unknown Stopped: False
DEBUG:root:set pilot state to: Unknown
Adaptor specific modifications: fork
Working directory: /home/kkadam/agent
use standard proxy
Submit pilot job to: fork://localhost/
Pilot Job/BigJob URL:
bigjob:4d1ed72a-2111-11e1-bae6-a4badb0c3696:localhost State: Running
DEBUG:root:add subjob to queue of PJ:
bigjob:4d1ed72a-2111-11e1-bae6-a4badb0c3696:localhost
DEBUG:root:create dictionary for job description. Job-URL:
bigjob:4d1ed72a-2111-11e1-bae6-a4badb0c3696:localhost:jobs:4d8b0d96-2111-11e1-bae6-a4badb0c3696
DEBUG:root:initialized advert entry for job:
advert://advert.cct.lsu.edu:8080/bigjob/4d1ed72a-2111-11e1-bae6-a4badb0c3696/localhost/jobs/4d8b0d96-2111-11e1-bae6-a4badb0c3696
DEBUG:root:Set state of job:
advert://advert.cct.lsu.edu:8080/bigjob/4d1ed72a-2111-11e1-bae6-a4badb0c3696/localhost/jobs/4d8b0d96-2111-11e1-bae6-a4badb0c3696
to: Unknown
state: Unknown
state: Running
state: Running
state: Done
Cancel Pilot Job
stop pilot job: bigjob:4d1ed72a-2111-11e1-bae6-a4badb0c3696:localhost
DEBUG:root:create advert entry:
advert://advert.cct.lsu.edu:8080/bigjob/4d1ed72a-2111-11e1-bae6-a4badb0c3696/localhost
DEBUG:root:update state of pilot job to: Done Stopped: True
DEBUG:root:delete pilot job:
bigjob:4d1ed72a-2111-11e1-bae6-a4badb0c3696:localhost
delete job: bigjob:4d1ed72a-2111-11e1-bae6-a4badb0c3696:localhost:jobs:4d8b0d96-2111-11e1-bae6-a4badb0c3696
Cancel Pilot Job
stop pilot job: bigjob:4d1ed72a-2111-11e1-bae6-a4badb0c3696:localhost
DEBUG:root:delete pilot job:
bigjob:4d1ed72a-2111-11e1-bae6-a4badb0c3696:localhost



Thanks,
-Kundan

On Wed, Dec 7, 2011 at 12:05 PM, pradeep kumar Mantha
<pradeepm66 at gmail.com> wrote:
> HI!
>
> Your files content looks correct.  I think the  problem is still does occur
> because you are still pointing to sqlite3.
>    DEBUG:root:init BigJob w/: advert://localhost/?dbtype=sqlite3
>
> Please edit the example script to point to COORDINATION_URL to use
> advert.cct.lsu.edu and 8080 port.
>
> thanks
> pradeep
>
>
> On Wed, Dec 7, 2011 at 11:50 AM, Kundan Kadam <kkadam1 at tigers.lsu.edu>
> wrote:
>>
>> Sorry, I forgot to do adaptor configuration. Too many machines!! I'm
>> attaching new errors along with the files. I'll try it on sierra as
>> india gave similar errors (and it's down).
>>
>> Start Pilot Job/BigJob at: fork://localhost
>> DEBUG:root:init BigJob w/: advert://localhost/?dbtype=sqlite3
>> DEBUG:root:['/home/kkadam/../', '/home/kkadam',
>>
>> '/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/BigJob-0.3.2-py2.7.egg',
>>
>> '/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/redis-2.2.4-py2.7.egg',
>>
>> '/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/virtualenv-1.6.4-py2.7.egg',
>>
>> '/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/threadpool-1.2.7-py2.7.egg',
>>
>> '/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/uuid-1.30-py2.7.egg',
>>
>> '/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/redis-2.2.4-py2.7.egg',
>>
>> '/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/threadpool-1.2.7-py2.7.egg',
>>
>> '/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/uuid-1.30-py2.7.egg',
>>
>> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg',
>> '/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages',
>> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/2.7/site-packages',
>> '/home/kkadam',
>> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python27.zip',
>> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python2.7',
>> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python2.7/plat-linux2',
>> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python2.7/lib-tk',
>> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python2.7/lib-old',
>> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python2.7/lib-dynload',
>>
>> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python2.7/site-packages',
>> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/2.7/site-packages',
>> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/2.7/site-packages',
>>
>> '/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/BigJob-0.3.2-py2.7.egg/bigjob']
>> DEBUG:root:Utilizing ADVERT Backend
>> DEBUG:root:Parsing URL: advert://localhost/?dbtype=sqlite3
>> DEBUG:root:Server: localhost Port None server_connect_url: None
>> DEBUG:root:initialized BigJob: bigjob:c17323dc-20f9-11e1-94f1-a4badb0c3696
>> DEBUG:root:create pilot job entry on backend server:
>> bigjob:c17323dc-20f9-11e1-94f1-a4badb0c3696:localhost
>> DEBUG:root:create advert entry:
>> advert://localhost/bigjob/c17323dc-20f9-11e1-94f1-a4badb0c3696/localhost
>> Traceback (most recent call last):
>>  File "example_local_single.py", line 113, in <module>
>>    main()
>>  File "example_local_single.py", line 79, in main
>>    processes_per_node)
>>  File
>> "/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/BigJob-0.3.2-py2.7.egg/bigjob/bigjob_manager.py",
>> line 154, in start_pilot_job
>>    self.coordination.set_pilot_state(self.pilot_url,
>> str(saga.job.Unknown), False)
>>  File
>> "/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/BigJob-0.3.2-py2.7.egg/coordination/bigjob_coordination_advert.py",
>> line 67, in set_pilot_state
>>    pilot_dir = saga.advert.directory(saga.url(pilot_url),
>> saga.advert.Create | saga.advert.CreateParents |
>> saga.advert.ReadWrite)
>> saga.no_success: SAGA(NoSuccess): default_advert: Unable to establish
>> connection to database_connection: dbname='advertdb' host='localhost'
>> password='SAGA_client' port='5432' user='SAGA' Cannot establish
>> connection to the database: could not connect to server: Connection
>> refused
>>        Is the server running on host "localhost" and accepting
>>        TCP/IP connections on port 5432?
>>
>>
>> Cancel Pilot Job
>> stop pilot job: bigjob:c17323dc-20f9-11e1-94f1-a4badb0c3696:localhost
>> DEBUG:root:delete pilot job:
>> bigjob:c17323dc-20f9-11e1-94f1-a4badb0c3696:localhost
>>
>>
>> Thanks,
>> -Kundan
>>
>> On Wed, Dec 7, 2011 at 11:22 AM, pradeep kumar Mantha
>> <pmanth2 at tigers.lsu.edu> wrote:
>> > Hi!
>> >
>> >   Could you please let us know what coordination URL you are using in
>> > the
>> > example.
>> >
>> >   If you followed the  Advert adaptor configuration under bigjob
>> > tutorial.
>> >    Please let us know what is the content of $HOME/.saga.ini
>> > & $HOME/saga_adaptors/saga_adaptor_default_advert.ini
>> >
>> > There is a problem accessing advert service from compute nodes on alamo.
>> > A
>> > ticket has been raised for Futuregrid.  So I suggest you to use Sierra
>> > or
>> > India in the mean while.
>> >
>> > thanks
>> > pradeep.
>> >
>> >
>> >
>> >
>> >
>> > On Wed, Dec 7, 2011 at 11:04 AM, Kundan Kadam <kkadam1 at tigers.lsu.edu>
>> > wrote:
>> >>
>> >> Thanks all for prompt reply!! I'll try to keep SAGA and BIGJOB
>> >> questions only to respective mailing-list from now on.
>> >>
>> >> SAGA
>> >> I'll install python-boost on my account. I'll get in touch with you in
>> >> case of any problems. Thanks again!!
>> >>
>> >>
>> >> BIGJOB
>> >> I'm on alamo. I load SAGA in my environment and run the example file,
>> >> output is as follows.
>> >>
>> >> $ /N/soft/SAGA/external/python/2.7.1/gcc-4.1.2//bin/python
>> >> example_local_single.py
>> >>
>> >> Start Pilot Job/BigJob at: fork://localhost
>> >> DEBUG:root:init BigJob w/: advert://localhost/?dbtype=sqlite3
>> >> DEBUG:root:['/home/kkadam/../', '/home/kkadam',
>> >>
>> >>
>> >> '/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/BigJob-0.3.2-py2.7.egg',
>> >>
>> >>
>> >> '/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/redis-2.2.4-py2.7.egg',
>> >>
>> >>
>> >> '/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/virtualenv-1.6.4-py2.7.egg',
>> >>
>> >>
>> >> '/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/threadpool-1.2.7-py2.7.egg',
>> >>
>> >>
>> >> '/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/uuid-1.30-py2.7.egg',
>> >>
>> >>
>> >> '/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/redis-2.2.4-py2.7.egg',
>> >>
>> >>
>> >> '/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/threadpool-1.2.7-py2.7.egg',
>> >>
>> >>
>> >> '/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/uuid-1.30-py2.7.egg',
>> >>
>> >>
>> >> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg',
>> >> '/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages',
>> >> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/2.7/site-packages',
>> >> '/home/kkadam',
>> >> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python27.zip',
>> >> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python2.7',
>> >>
>> >> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python2.7/plat-linux2',
>> >> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python2.7/lib-tk',
>> >> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python2.7/lib-old',
>> >>
>> >> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python2.7/lib-dynload',
>> >>
>> >>
>> >> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/python2.7/site-packages',
>> >> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/2.7/site-packages',
>> >> '/N/soft/SAGA/external/python/2.7.1/gcc-4.1.2/lib/2.7/site-packages',
>> >>
>> >>
>> >> '/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/BigJob-0.3.2-py2.7.egg/bigjob']
>> >> DEBUG:root:Utilizing ADVERT Backend
>> >> DEBUG:root:Parsing URL: advert://localhost/?dbtype=sqlite3
>> >> DEBUG:root:Server: localhost Port None server_connect_url: None
>> >> DEBUG:root:initialized BigJob:
>> >> bigjob:eee22e8a-20f4-11e1-ad65-a4badb0c3696
>> >> DEBUG:root:create pilot job entry on backend server:
>> >> bigjob:eee22e8a-20f4-11e1-ad65-a4badb0c3696:localhost
>> >> DEBUG:root:create advert entry:
>> >>
>> >> advert://localhost/bigjob/eee22e8a-20f4-11e1-ad65-a4badb0c3696/localhost
>> >> Traceback (most recent call last):
>> >>  File "example_local_single.py", line 113, in <module>
>> >>    main()
>> >>  File "example_local_single.py", line 79, in main
>> >>    processes_per_node)
>> >>  File
>> >>
>> >> "/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/BigJob-0.3.2-py2.7.egg/bigjob/bigjob_manager.py",
>> >> line 154, in start_pilot_job
>> >>    self.coordination.set_pilot_state(self.pilot_url,
>> >> str(saga.job.Unknown), False)
>> >>  File
>> >>
>> >> "/N/soft/SAGA/saga/1.6/gcc-4.1.2/lib/python2.7/site-packages/BigJob-0.3.2-py2.7.egg/coordination/bigjob_coordination_advert.py",
>> >> line 67, in set_pilot_state
>> >>    pilot_dir = saga.advert.directory(saga.url(pilot_url),
>> >> saga.advert.Create | saga.advert.CreateParents |
>> >> saga.advert.ReadWrite)
>> >> saga.no_success: SAGA(NoSuccess): default_advert: Unable to establish
>> >> connection to database_connection: dbname='advertdb' host='localhost'
>> >> password='<password>' port='5432' user='<username>' Cannot establish
>> >> connection to the database: could not connect to server: Connection
>> >> refused
>> >>        Is the server running on host "localhost" and accepting
>> >>        TCP/IP connections on port 5432?
>> >>
>> >>
>> >> Cancel Pilot Job
>> >> stop pilot job: bigjob:eee22e8a-20f4-11e1-ad65-a4badb0c3696:localhost
>> >> DEBUG:root:delete pilot job:
>> >> bigjob:eee22e8a-20f4-11e1-ad65-a4badb0c3696:localhost
>> >>
>> >>
>> >>
>> >> Thanks,
>> >> -Kundan
>> >>
>> >>
>> >>
>> >>
>> >> On Tue, Dec 6, 2011 at 2:17 PM, Melissa Romanus
>> >> <melissa.romanus at rutgers.edu> wrote:
>> >> > Hi Andre,
>> >> >
>> >> > This topic interests me as well. Are there tutorials for using ssh to
>> >> > launch bigjob on the website?
>> >> >
>> >> > -Melissa
>> >> >
>> >> > On Tue, Dec 6, 2011 at 2:59 PM, Andre Luckow <andre.luckow at gmail.com>
>> >> > wrote:
>> >> >> Hi Kundan,
>> >> >> Can you sent the complete error. Are you using ssh to launch BJ;
>> >> >> then
>> >> >> you
>> >> >> need to setup certificate based authentication.
>> >> >>
>> >> >> Best,
>> >> >> Andre
>> >> >>
>> >> >>
>> >> >> On Tuesday, December 6, 2011, Kundan Kadam <kkadam1 at tigers.lsu.edu>
>> >> >> wrote:
>> >> >>> This is embarrassing but I am having trouble installing SAGA
>> >> >>> locally
>> >> >>> on any of the Futuregrid machines. I could only install SAGA core
>> >> >>> on
>> >> >>> hotel, but none of the python bindings or adapters. Problem is
>> >> >>> location of python, even when it is apparently correct, there's an
>> >> >>> error saying boost python flags/ headers not found. Am I missing
>> >> >>> something obvious or should I contact Futuregrid support?
>> >> >>>
>> >> >>> However I could get the the preinstalled SAGA loaded into my
>> >> >>> environment.
>> >> >>> (using
>> >> >>> http://www.saga-project.org/documentation/deployment/xsede-fg) I
>> >> >>> got
>> >> >>> the BIGJOB installed.  (Using method b of
>> >> >>> http://faust.cct.lsu.edu/trac/bigjob/wiki/install) And apparently
>> >> >>> it
>> >> >>> works because I can say import saga and import bigjob in python
>> >> >>> without errors.
>> >> >>>
>> >> >>> Problem is running examples. (example_local_single.py etc) The
>> >> >>> tutorial (http://faust.cct.lsu.edu/trac/bigjob/wiki/Tutorials) says
>> >> >>> just to run it without modifying the file. But it gives an error
>> >> >>> saying "Are you sure you have access to localhost?" or similar. Do
>> >> >>> I
>> >> >>> need to configure the framework before that somehow?
>> >> >>>
>> >> >>> At the moment the machines are down for monthly maintainance.
>> >> >>>
>> >> >>> Thanks,
>> >> >>> -Kundan
>> >> >>> _______________________________________________
>> >> >>> saga-users mailing list
>> >> >>> saga-users at cct.lsu.edu
>> >> >>> https://mail.cct.lsu.edu/mailman/listinfo/saga-users
>> >> >>>
>> >> >>
>> >> >> _______________________________________________
>> >> >> saga-users mailing list
>> >> >> saga-users at cct.lsu.edu
>> >> >> https://mail.cct.lsu.edu/mailman/listinfo/saga-users
>> >> >>
>> >> _______________________________________________
>> >> Bigjob-users mailing list
>> >> Bigjob-users at mail.cct.lsu.edu
>> >> https://mail.cct.lsu.edu/mailman/listinfo/bigjob-users
>> >
>> >
>>
>> _______________________________________________
>> Bigjob-users mailing list
>> Bigjob-users at mail.cct.lsu.edu
>> https://mail.cct.lsu.edu/mailman/listinfo/bigjob-users
>>
>

More information about the Bigjob-users mailing list