[Bigjob-users] bigjob on teragrid/xsede

Wed Jan 4 16:36:00 CST 2012

I am on a very bad connection at the moment, and can't connect to the
machine.  I'll have a look at this in about 9 hours from now.

Thanks, Andre.

On Wed, Jan 4, 2012 at 11:27 PM, Yaakoub El Khamra <yelkhamra at gmail.com> wrote:
>
> So this is what we are doing.
>
> Andre M you have until 9:30 central tonight to update bigjob installations
> (easy_install -U bigjob) in the python installations of both ranger and
> lonestar CSA's. If you don't I am going to do it myself from your account.
>
> I fixed a minor permissions issue for the readme.
>
>
> Regards
> Yaakoub El Khamra
>
>
>
>
> On Wed, Jan 4, 2012 at 3:47 PM, Yaakoub El Khamra <yelkhamra at gmail.com>
> wrote:
>>
>>
>> Actually Andre this works. You just have to use a 3rd freaking python:
>>
>> which python
>>
>>
>> /scratch/projects/tg/SAGA/external/python/2.7.1/gcc-4.4.5/bin/python
>>
>>  python
>>
>>
>> Python 2.7.1 (r271:86832, Nov 25 2011, 04:51:00)
>> [GCC 4.4.5] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> from saga import *
>> >>>
>>
>> That's how I got it to work. Ok we need to clean this up. Easiest way to
>> do it would be for Andre M/L to install BigJob in the SAGA python with
>> easy_install and keep it up to date. We also need to make this uniform
>> across all machines.
>>
>>
>> Regards
>> Yaakoub El Khamra
>>
>>
>>
>>
>> On Wed, Jan 4, 2012 at 3:42 PM, Yaakoub El Khamra <yelkhamra at gmail.com>
>> wrote:
>>>
>>>
>>> Andre M, your thoughts?
>>>
>>> This was working late nov for me. What happened?
>>>
>>> Regards
>>> Yaakoub El Khamra
>>>
>>>
>>>
>>>
>>> On Wed, Jan 4, 2012 at 3:38 PM, Andre Luckow <aluckow at cct.lsu.edu> wrote:
>>>>
>>>> Hi Yaakoub,
>>>> we have SAGA and Python in CSA space that works fine with BJ:
>>>>
>>>>
>>>> https://svn.cct.lsu.edu/repos/saga-projects/deployment/tg-csa/doc/README.saga-1.6.gcc-4.4.5.ranger
>>>>
>>>> Unfortunately, it's broken :-( - will open a ticket:
>>>>
>>>> tg804093 at login3 ~$ python
>>>> Python 2.7.1 (r271:86832, Nov 25 2011, 04:51:00)
>>>> [GCC 4.4.5] on linux2
>>>> Type "help", "copyright", "credits" or "license" for more information.
>>>> >>> import saga
>>>> Traceback (most recent call last):
>>>>  File "<stdin>", line 1, in <module>
>>>>  File
>>>> "/scratch/projects/tg/SAGA/saga/1.6/gcc-4.4.5/lib/python2.7/site-packages/saga/__init__.py",
>>>> line 28, in <module>
>>>>    from _engine import *         # import all classes from SAGA.engine
>>>> ImportError: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.9' not
>>>> found (required by
>>>>
>>>> /scratch/projects/tg/SAGA/saga/1.6/gcc-4.4.5/lib/libsaga_package_advert.so.1.6)
>>>>
>>>> Best,
>>>> Andre
>>>>
>>>> On Wed, Jan 4, 2012 at 10:28 PM, Yaakoub El Khamra <yelkhamra at gmail.com>
>>>> wrote:
>>>> >
>>>> > python bigjob-bootstrap.py $WORK/ECSS/bigjob_install
>>>> >
>>>> >      Traceback (most recent call last):
>>>> >   File "bigjob-bootstrap.py", line 23, in ?
>>>> >     import subprocess
>>>> > ImportError: No module named subprocess
>>>> >
>>>> > SAGA on ranger was compiled with python 2.3.4, the system python.
>>>> > Andre M,
>>>> > j'accuse!
>>>> >
>>>> > You have to load the python module to get bigjob. This can be better
>>>> > coordinated IMHO.
>>>> >
>>>> > Regards
>>>> > Yaakoub El Khamra
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Wed, Jan 4, 2012 at 3:22 PM, Andre Luckow <aluckow at cct.lsu.edu>
>>>> > wrote:
>>>> >>
>>>> >> Hi Yaakoub,
>>>> >> the bootstrap script in variant c) installs virtualenv automatically
>>>> >> for the user. Option d) should work. We just migrated to Github
>>>> >> yesterday - so there might be some issues. Not sure why your git
>>>> >> version doesn't accept the Github certificate. But, you can also
>>>> >> clone
>>>> >> the repository via HTTP.
>>>> >>
>>>> >> git clone http://github.com/drelu/BigJob.git
>>>> >>
>>>> >> or the native git protocol
>>>> >>
>>>> >> git clone git://github.com/drelu/BigJob.git
>>>> >>
>>>> >> Or alternatively, the good old SVN.
>>>> >>
>>>> >> I think we have many options to install BJ (maybe to many for a
>>>> >> normal
>>>> >> user). I personally prefer the virtualenv alternative since this is a
>>>> >> clean way to manage all your Python libraries in user space.
>>>> >>
>>>> >> Best,
>>>> >> Andre
>>>> >>
>>>> >>
>>>> >> On Wed, Jan 4, 2012 at 10:02 PM, Yaakoub El Khamra
>>>> >> <yelkhamra at gmail.com>
>>>> >> wrote:
>>>> >> >
>>>> >> > Also can I point out something I should have pointed out a while
>>>> >> > ago:
>>>> >> > users have to install virtualenv on almost all supercomputers. At
>>>> >> > least
>>>> >> > the
>>>> >> > top 5 on XSEDE, all the ones we have allocations on.
>>>> >> >
>>>> >> > This means, users have to install virtualenv to install bigjob.
>>>> >> > That's
>>>> >> > not
>>>> >> > practical. d) is what most users will want to use since they
>>>> >> > already
>>>> >> > have to
>>>> >> > add to the pythonpath from the saga installation.
>>>> >> >
>>>> >> > Regards
>>>> >> > Yaakoub El Khamra
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > On Wed, Jan 4, 2012 at 2:39 PM, Yaakoub El Khamra
>>>> >> > <yelkhamra at gmail.com>
>>>> >> > wrote:
>>>> >> >>
>>>> >> >> Cloning into BigJob...
>>>> >> >> error: SSL certificate problem, verify that the CA cert is OK.
>>>> >> >> Details:
>>>> >> >> error:14090086:SSL
>>>> >> >> routines:SSL3_GET_SERVER_CERTIFICATE:certificate
>>>> >> >> verify
>>>> >> >> failed while accessing
>>>> >> >> https://github.com/drelu/BigJob.git/info/refs
>>>> >> >>
>>>> >> >> fatal: HTTP request failed
>>>> >> >>
>>>> >> >> Regards
>>>> >> >> Yaakoub El Khamra
>>>> >> >>
>>>> >> >>
>>>> >> >>
>>>> >> >>
>>>> >> >> On Wed, Jan 4, 2012 at 2:33 PM, Andre Luckow <aluckow at cct.lsu.edu>
>>>> >> >> wrote:
>>>> >> >>>
>>>> >> >>> Hi Yaakoub,
>>>> >> >>> with what error does d) fail for you?
>>>> >> >>>
>>>> >> >>> Best,
>>>> >> >>> Andre
>>>> >> >>>
>>>> >> >>> On Wed, Jan 4, 2012 at 9:18 PM, Yaakoub El Khamra
>>>> >> >>> <yelkhamra at gmail.com>
>>>> >> >>> wrote:
>>>> >> >>> >
>>>> >> >>> > FYI d installation method fails. Sorry I should have told you
>>>> >> >>> > guys
>>>> >> >>> > about
>>>> >> >>> > this a very long time ago when I first discovered it.
>>>> >> >>> >
>>>> >> >>> > Regards
>>>> >> >>> > Yaakoub El Khamra
>>>> >> >>> >
>>>> >> >>> >
>>>> >> >>> >
>>>> >> >>> >
>>>> >> >>> > On Wed, Jan 4, 2012 at 3:03 AM, Andre Luckow
>>>> >> >>> > <aluckow at cct.lsu.edu>
>>>> >> >>> > wrote:
>>>> >> >>> >>
>>>> >> >>> >> Hi Melissa,
>>>> >> >>> >> the recommended way is to install BJ in user space. The CSA BJ
>>>> >> >>> >> version
>>>> >> >>> >> is a bit outdated and is already known to have a couple of
>>>> >> >>> >> bugs:
>>>> >> >>> >>
>>>> >> >>> >> http://faust.cct.lsu.edu/trac/bigjob/wiki/install
>>>> >> >>> >>
>>>> >> >>> >> Installation type b or c should be the best for you.
>>>> >> >>> >>
>>>> >> >>> >> Once you have setup your virtualenv, you can easily update to
>>>> >> >>> >> a new
>>>> >> >>> >> version of BJ with:
>>>> >> >>> >>
>>>> >> >>> >> easy_install -U bigjob
>>>> >> >>> >>
>>>> >> >>> >> Since, I expect that we will discover some bugs on the way to
>>>> >> >>> >> XSEDE,
>>>> >> >>> >> this will be an important feature.
>>>> >> >>> >>
>>>> >> >>> >> Best,
>>>> >> >>> >> Andre
>>>> >> >>> >>
>>>> >> >>> >> On Wed, Jan 4, 2012 at 4:24 AM, Melissa Romanus
>>>> >> >>> >> <melissa.romanus at rutgers.edu> wrote:
>>>> >> >>> >> > Hi all-
>>>> >> >>> >> >
>>>> >> >>> >> > I need to start either compiling information or playing
>>>> >> >>> >> > around
>>>> >> >>> >> > with
>>>> >> >>> >> > bigjob
>>>> >> >>> >> > on xsede/teragrid, specifically kraken, ranger, and possibly
>>>> >> >>> >> > lonestar.
>>>> >> >>> >> > Last
>>>> >> >>> >> > semester, I was only working on the futuregrid. What
>>>> >> >>> >> > documentation
>>>> >> >>> >> > currently
>>>> >> >>> >> > exists for getting bj running on the XSEDE machines? Also, I
>>>> >> >>> >> > am
>>>> >> >>> >> > sorry I
>>>> >> >>> >> > have
>>>> >> >>> >> > not been super up on email traffic, but what is the general
>>>> >> >>> >> > consensus of
>>>> >> >>> >> > 'how to use bj now' (in other words, install into user space
>>>> >> >>> >> > vs
>>>> >> >>> >> > CSA)? I
>>>> >> >>> >> > am
>>>> >> >>> >> > asking because I will soon be working closely with the
>>>> >> >>> >> > chemists
>>>> >> >>> >> > and
>>>> >> >>> >> > want
>>>> >> >>> >> > to
>>>> >> >>> >> > have an easy/smooth process for them to follow.
>>>> >> >>> >> >
>>>> >> >>> >> > Thank you,
>>>> >> >>> >> > Melissa
>>>> >> >>> >> >
>>>> >> >>> >> > _______________________________________________
>>>> >> >>> >> > Bigjob-users mailing list
>>>> >> >>> >> > Bigjob-users at mail.cct.lsu.edu
>>>> >> >>> >> > https://mail.cct.lsu.edu/mailman/listinfo/bigjob-users
>>>> >> >>> >> >
>>>> >> >>> >> _______________________________________________
>>>> >> >>> >> Bigjob-users mailing list
>>>> >> >>> >> Bigjob-users at mail.cct.lsu.edu
>>>> >> >>> >> https://mail.cct.lsu.edu/mailman/listinfo/bigjob-users
>>>> >> >>> >
>>>> >> >>> >
>>>> >> >>
>>>> >> >>
>>>> >> >
>>>> >
>>>> >
>>>
>>>
>>
>

-- 
Nothing is ever easy...