[Bigjob-users] bigjob on teragrid/xsede

Wed Jan 4 16:45:08 CST 2012

Just answer me this: how did you install bigjob? Did you do an easy_install
bigjob?

Yes or no, simple question there Andre.

Regards
Yaakoub El Khamra

On Wed, Jan 4, 2012 at 4:36 PM, Andre Merzky <andre at merzky.net> wrote:

> I am on a very bad connection at the moment, and can't connect to the
> machine.  I'll have a look at this in about 9 hours from now.
>
> Thanks, Andre.
>
>
> On Wed, Jan 4, 2012 at 11:27 PM, Yaakoub El Khamra <yelkhamra at gmail.com>
> wrote:
> >
> > So this is what we are doing.
> >
> > Andre M you have until 9:30 central tonight to update bigjob
> installations
> > (easy_install -U bigjob) in the python installations of both ranger and
> > lonestar CSA's. If you don't I am going to do it myself from your
> account.
> >
> > I fixed a minor permissions issue for the readme.
> >
> >
> > Regards
> > Yaakoub El Khamra
> >
> >
> >
> >
> > On Wed, Jan 4, 2012 at 3:47 PM, Yaakoub El Khamra <yelkhamra at gmail.com>
> > wrote:
> >>
> >>
> >> Actually Andre this works. You just have to use a 3rd freaking python:
> >>
> >> which python
> >>
> >>
> >> /scratch/projects/tg/SAGA/external/python/2.7.1/gcc-4.4.5/bin/python
> >>
> >>  python
> >>
> >>
> >> Python 2.7.1 (r271:86832, Nov 25 2011, 04:51:00)
> >> [GCC 4.4.5] on linux2
> >> Type "help", "copyright", "credits" or "license" for more information.
> >> >>> from saga import *
> >> >>>
> >>
> >> That's how I got it to work. Ok we need to clean this up. Easiest way to
> >> do it would be for Andre M/L to install BigJob in the SAGA python with
> >> easy_install and keep it up to date. We also need to make this uniform
> >> across all machines.
> >>
> >>
> >> Regards
> >> Yaakoub El Khamra
> >>
> >>
> >>
> >>
> >> On Wed, Jan 4, 2012 at 3:42 PM, Yaakoub El Khamra <yelkhamra at gmail.com>
> >> wrote:
> >>>
> >>>
> >>> Andre M, your thoughts?
> >>>
> >>> This was working late nov for me. What happened?
> >>>
> >>> Regards
> >>> Yaakoub El Khamra
> >>>
> >>>
> >>>
> >>>
> >>> On Wed, Jan 4, 2012 at 3:38 PM, Andre Luckow <aluckow at cct.lsu.edu>
> wrote:
> >>>>
> >>>> Hi Yaakoub,
> >>>> we have SAGA and Python in CSA space that works fine with BJ:
> >>>>
> >>>>
> >>>>
> https://svn.cct.lsu.edu/repos/saga-projects/deployment/tg-csa/doc/README.saga-1.6.gcc-4.4.5.ranger
> >>>>
> >>>> Unfortunately, it's broken :-( - will open a ticket:
> >>>>
> >>>> tg804093 at login3 ~$ python
> >>>> Python 2.7.1 (r271:86832, Nov 25 2011, 04:51:00)
> >>>> [GCC 4.4.5] on linux2
> >>>> Type "help", "copyright", "credits" or "license" for more information.
> >>>> >>> import saga
> >>>> Traceback (most recent call last):
> >>>>  File "<stdin>", line 1, in <module>
> >>>>  File
> >>>>
> "/scratch/projects/tg/SAGA/saga/1.6/gcc-4.4.5/lib/python2.7/site-packages/saga/__init__.py",
> >>>> line 28, in <module>
> >>>>    from _engine import *         # import all classes from SAGA.engine
> >>>> ImportError: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.9' not
> >>>> found (required by
> >>>>
> >>>>
> /scratch/projects/tg/SAGA/saga/1.6/gcc-4.4.5/lib/libsaga_package_advert.so.1.6)
> >>>>
> >>>> Best,
> >>>> Andre
> >>>>
> >>>> On Wed, Jan 4, 2012 at 10:28 PM, Yaakoub El Khamra <
> yelkhamra at gmail.com>
> >>>> wrote:
> >>>> >
> >>>> > python bigjob-bootstrap.py $WORK/ECSS/bigjob_install
> >>>> >
> >>>> >      Traceback (most recent call last):
> >>>> >   File "bigjob-bootstrap.py", line 23, in ?
> >>>> >     import subprocess
> >>>> > ImportError: No module named subprocess
> >>>> >
> >>>> > SAGA on ranger was compiled with python 2.3.4, the system python.
> >>>> > Andre M,
> >>>> > j'accuse!
> >>>> >
> >>>> > You have to load the python module to get bigjob. This can be better
> >>>> > coordinated IMHO.
> >>>> >
> >>>> > Regards
> >>>> > Yaakoub El Khamra
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > On Wed, Jan 4, 2012 at 3:22 PM, Andre Luckow <aluckow at cct.lsu.edu>
> >>>> > wrote:
> >>>> >>
> >>>> >> Hi Yaakoub,
> >>>> >> the bootstrap script in variant c) installs virtualenv
> automatically
> >>>> >> for the user. Option d) should work. We just migrated to Github
> >>>> >> yesterday - so there might be some issues. Not sure why your git
> >>>> >> version doesn't accept the Github certificate. But, you can also
> >>>> >> clone
> >>>> >> the repository via HTTP.
> >>>> >>
> >>>> >> git clone http://github.com/drelu/BigJob.git
> >>>> >>
> >>>> >> or the native git protocol
> >>>> >>
> >>>> >> git clone git://github.com/drelu/BigJob.git
> >>>> >>
> >>>> >> Or alternatively, the good old SVN.
> >>>> >>
> >>>> >> I think we have many options to install BJ (maybe to many for a
> >>>> >> normal
> >>>> >> user). I personally prefer the virtualenv alternative since this
> is a
> >>>> >> clean way to manage all your Python libraries in user space.
> >>>> >>
> >>>> >> Best,
> >>>> >> Andre
> >>>> >>
> >>>> >>
> >>>> >> On Wed, Jan 4, 2012 at 10:02 PM, Yaakoub El Khamra
> >>>> >> <yelkhamra at gmail.com>
> >>>> >> wrote:
> >>>> >> >
> >>>> >> > Also can I point out something I should have pointed out a while
> >>>> >> > ago:
> >>>> >> > users have to install virtualenv on almost all supercomputers. At
> >>>> >> > least
> >>>> >> > the
> >>>> >> > top 5 on XSEDE, all the ones we have allocations on.
> >>>> >> >
> >>>> >> > This means, users have to install virtualenv to install bigjob.
> >>>> >> > That's
> >>>> >> > not
> >>>> >> > practical. d) is what most users will want to use since they
> >>>> >> > already
> >>>> >> > have to
> >>>> >> > add to the pythonpath from the saga installation.
> >>>> >> >
> >>>> >> > Regards
> >>>> >> > Yaakoub El Khamra
> >>>> >> >
> >>>> >> >
> >>>> >> >
> >>>> >> >
> >>>> >> > On Wed, Jan 4, 2012 at 2:39 PM, Yaakoub El Khamra
> >>>> >> > <yelkhamra at gmail.com>
> >>>> >> > wrote:
> >>>> >> >>
> >>>> >> >> Cloning into BigJob...
> >>>> >> >> error: SSL certificate problem, verify that the CA cert is OK.
> >>>> >> >> Details:
> >>>> >> >> error:14090086:SSL
> >>>> >> >> routines:SSL3_GET_SERVER_CERTIFICATE:certificate
> >>>> >> >> verify
> >>>> >> >> failed while accessing
> >>>> >> >> https://github.com/drelu/BigJob.git/info/refs
> >>>> >> >>
> >>>> >> >> fatal: HTTP request failed
> >>>> >> >>
> >>>> >> >> Regards
> >>>> >> >> Yaakoub El Khamra
> >>>> >> >>
> >>>> >> >>
> >>>> >> >>
> >>>> >> >>
> >>>> >> >> On Wed, Jan 4, 2012 at 2:33 PM, Andre Luckow <
> aluckow at cct.lsu.edu>
> >>>> >> >> wrote:
> >>>> >> >>>
> >>>> >> >>> Hi Yaakoub,
> >>>> >> >>> with what error does d) fail for you?
> >>>> >> >>>
> >>>> >> >>> Best,
> >>>> >> >>> Andre
> >>>> >> >>>
> >>>> >> >>> On Wed, Jan 4, 2012 at 9:18 PM, Yaakoub El Khamra
> >>>> >> >>> <yelkhamra at gmail.com>
> >>>> >> >>> wrote:
> >>>> >> >>> >
> >>>> >> >>> > FYI d installation method fails. Sorry I should have told you
> >>>> >> >>> > guys
> >>>> >> >>> > about
> >>>> >> >>> > this a very long time ago when I first discovered it.
> >>>> >> >>> >
> >>>> >> >>> > Regards
> >>>> >> >>> > Yaakoub El Khamra
> >>>> >> >>> >
> >>>> >> >>> >
> >>>> >> >>> >
> >>>> >> >>> >
> >>>> >> >>> > On Wed, Jan 4, 2012 at 3:03 AM, Andre Luckow
> >>>> >> >>> > <aluckow at cct.lsu.edu>
> >>>> >> >>> > wrote:
> >>>> >> >>> >>
> >>>> >> >>> >> Hi Melissa,
> >>>> >> >>> >> the recommended way is to install BJ in user space. The CSA
> BJ
> >>>> >> >>> >> version
> >>>> >> >>> >> is a bit outdated and is already known to have a couple of
> >>>> >> >>> >> bugs:
> >>>> >> >>> >>
> >>>> >> >>> >> http://faust.cct.lsu.edu/trac/bigjob/wiki/install
> >>>> >> >>> >>
> >>>> >> >>> >> Installation type b or c should be the best for you.
> >>>> >> >>> >>
> >>>> >> >>> >> Once you have setup your virtualenv, you can easily update
> to
> >>>> >> >>> >> a new
> >>>> >> >>> >> version of BJ with:
> >>>> >> >>> >>
> >>>> >> >>> >> easy_install -U bigjob
> >>>> >> >>> >>
> >>>> >> >>> >> Since, I expect that we will discover some bugs on the way
> to
> >>>> >> >>> >> XSEDE,
> >>>> >> >>> >> this will be an important feature.
> >>>> >> >>> >>
> >>>> >> >>> >> Best,
> >>>> >> >>> >> Andre
> >>>> >> >>> >>
> >>>> >> >>> >> On Wed, Jan 4, 2012 at 4:24 AM, Melissa Romanus
> >>>> >> >>> >> <melissa.romanus at rutgers.edu> wrote:
> >>>> >> >>> >> > Hi all-
> >>>> >> >>> >> >
> >>>> >> >>> >> > I need to start either compiling information or playing
> >>>> >> >>> >> > around
> >>>> >> >>> >> > with
> >>>> >> >>> >> > bigjob
> >>>> >> >>> >> > on xsede/teragrid, specifically kraken, ranger, and
> possibly
> >>>> >> >>> >> > lonestar.
> >>>> >> >>> >> > Last
> >>>> >> >>> >> > semester, I was only working on the futuregrid. What
> >>>> >> >>> >> > documentation
> >>>> >> >>> >> > currently
> >>>> >> >>> >> > exists for getting bj running on the XSEDE machines?
> Also, I
> >>>> >> >>> >> > am
> >>>> >> >>> >> > sorry I
> >>>> >> >>> >> > have
> >>>> >> >>> >> > not been super up on email traffic, but what is the
> general
> >>>> >> >>> >> > consensus of
> >>>> >> >>> >> > 'how to use bj now' (in other words, install into user
> space
> >>>> >> >>> >> > vs
> >>>> >> >>> >> > CSA)? I
> >>>> >> >>> >> > am
> >>>> >> >>> >> > asking because I will soon be working closely with the
> >>>> >> >>> >> > chemists
> >>>> >> >>> >> > and
> >>>> >> >>> >> > want
> >>>> >> >>> >> > to
> >>>> >> >>> >> > have an easy/smooth process for them to follow.
> >>>> >> >>> >> >
> >>>> >> >>> >> > Thank you,
> >>>> >> >>> >> > Melissa
> >>>> >> >>> >> >
> >>>> >> >>> >> > _______________________________________________
> >>>> >> >>> >> > Bigjob-users mailing list
> >>>> >> >>> >> > Bigjob-users at mail.cct.lsu.edu
> >>>> >> >>> >> > https://mail.cct.lsu.edu/mailman/listinfo/bigjob-users
> >>>> >> >>> >> >
> >>>> >> >>> >> _______________________________________________
> >>>> >> >>> >> Bigjob-users mailing list
> >>>> >> >>> >> Bigjob-users at mail.cct.lsu.edu
> >>>> >> >>> >> https://mail.cct.lsu.edu/mailman/listinfo/bigjob-users
> >>>> >> >>> >
> >>>> >> >>> >
> >>>> >> >>
> >>>> >> >>
> >>>> >> >
> >>>> >
> >>>> >
> >>>
> >>>
> >>
> >
>
>
>
> --
> Nothing is ever easy...
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cct.lsu.edu/pipermail/bigjob-users/attachments/20120104/ce04c7a5/attachment-0001.html 