[Saga-devel] Re: aws / ssh adaptors compilation problems (fwd)

[Andre Merzky]

Quoting [Katerina Stamou] (Jan 12 2009):
> 
> Dear Shantenu, Andre,
> 
> As we are coming short of time, a quick update to where I am right now 
> and a suggestion for a possible chat tomorrow.
> 
> I've managed to start the master process in the image and right now I am 
> trying to make MapReduce run with cloud instances. Actually, one has to 
> install boost and the rest of the saga libraries on the vm and of course 
> the agent program compiled with those in order to be able to actually 
> submit sth. The default image in Eucaliptus installation is really 
> minimal (4MB) and doesn't provide space for extra libs. 

You mean, there is really only 4 MB extra space??  That does
not even allow us to use saga-lite (which is about 150 MB,
and additionally requires boost etc).

As for deployment: the saga.ini for the aws adaptors
contains a key 

  ec2_image_prep = $[saga.adaptor_suite.aws.preferences.ec2_scripts]/saga-image-prep-ec2

which points to a script which is executed on each newly
started VM instance.  You can add a couple commands there
which would deploy boost and saga.  E.g.:

jonas merzky ~/links/saga/trunk/adaptors/aws : cat ./internal/bin/saga-image-prep-ec2

  apt-get install sshfs
  apt-get install boost
  apt-get install openssh
  apt-get install postgresql-client
# wget [some location]/saga-bin.tgz 
  tar zxvf saga-bin.tgz
  mkdir -p /usr/local/packages
  mv saga-bin /usr/local/packages/saga

  ldconfig
  ldconfig /usr/local/packages/saga/lib/

I did not test the saga bin part in the automatic
deployment, but it works ok manually.  The apt-get commands
and ldconfig seem to work ok from within that script.

That increases the startup time for the VM of course - but
we need to measure that separately anyway, IMHO.


> BTW, is there 
> any common saga-account with the necessary credentials for Amazon's EC2? 

I have a private account there, on my own Credit Card.  But
also, I have a account on the Eucalyptus test cloud, which
would be great to use, too.  That should have more space for
deployment of software (although it has a firewall in place,
not sure if it gets apt-get happy, or if we need to stage
everything from macpro.cct, which is excempt from the
firewall).

I don't mind you using my amazon account too much, really -
but in case you accidentically deploy a large number of VMs
(almost happened to me once), this can get expensive... :-/

I'll send the credentials in a separate mail.


> If so, we can simply make a run there, instead of trying to deploy our 
> own image in Eucaliptus, which I m looking into right now, and of course 
> is more time consuming. 
> So, in terms of progress and further sections to be added in the paper 
> we are in a very good point, imho. I should clarify that I am working 
> with only one instance, and this due to the administrative problems in 
> GumboCloud, we were discussing with Archit last week. I don't know if  
> they will be able to solve the remaining issues by tomorrow, in order 
> for us to be able to use more instances. So, for the moment our test 
> will be one worker on LONI clusters and one worker on GumboCloud. I'll 
> continue working on it till late tonight, and I hope that our final goal 
> will be accomplished.

Pity that the one-instance limitation still exists!  Please
follow up with Archit on that.

Creating a new image should not be _that_ difficult, at
least it is not for EC2.  Could you also check with Archit
if that would be possible?

We would need the stuff from above: boost libs, postgresql
libs (for the advert adaptor), sshfs, ssh, and SAGA from
SVN.  It would be good to document the procedure somehow, in
case we need to update SAGA.  Also, some space (~500 MB) for
later software deployment would be useful, not to speak of
MapReduce data sets which need to be stored locally as well,
at times.

> 
> Thus, I propose to have a quick chat tomorrow afternoon through Skype to 
> discuss the rest of the details. I have classes from 12 to 13.30 and 
> 15.00 to 16.30 BTR time. It would be convenient if we arrange a chat 
> around 14.00 BTR, which would be around 20.00 and 21.00 o'clock for you. 
> Please let me know if this is feasible.

Works for me!

Cheers, Andre.


> I guess that's all for now,
> 
> Best,
> Katerina
> 
> Shantenu Jha wrote:
> >Hi Kate, Andre,
> >
> >No pressure, but let me know if you think it might be useful to
> >chat today. Else tomorrow is also OK.
> >
> >Cheers,
> >Shantenu
> >
> >---------- Forwarded message ----------
> >Date: Sun, 11 Jan 2009 10:57:55 +0100
> >From: Andre Merzky <andre at merzky.net>
> >To: Katerina Stamou <kstamou at cct.lsu.edu>
> >Cc: saga-devel at cct.lsu.edu
> >Subject: [Saga-devel] Re: aws / ssh adaptors compilation problems
> >
> >Hi Katarina,
> >
> >sorry, my commit seemed incomplete - utils.cpp is added now.
> >
> >Why object.hpp needs to be included I don't know - it is
> >included by impl/exception.hpp anyway (and that is where
> >object is used), but I added that fix in svn anyway.
> >
> >Thanks, Andre.
> >
> >
> >Quoting [Katerina Stamou] (Jan 11 2009):
> >>Date: Sun, 11 Jan 2009 03:00:52 -0600
> >>From: Katerina Stamou <kstamou at cct.lsu.edu>
> >>To: Andre Merzky <andre at merzky.net>, saga-devel at cct.lsu.edu
> >>Subject: aws / ssh adaptors compilation problems
> >>
> >>Hi Andre,
> >>
> >>I'm trying to get aws and ssh adaptors to compile and install for the
> >>eucalyptus mapreduce testing, and several problems come up with
> >>the last svn revision (3257):
> >>
> >>The compilation fails at two points with various warnings and errors
> >>related to:
> >>
> >>saga/saga/adaptors/utils/process/process.hpp
> >>and
> >>saga/saga/adaptors/utils/ in general
> >>
> >>I've managed to work around the first by adding:
> >>
> >>#include <saga/saga/object.hpp>
> >>
> >>to process.hpp.
> >>
> >>For the second problem, commenting out
> >>
> >>#SAGA_SRC            = utils.cpp
> >>#SAGA_OBJ            = utils.o
> >>
> >>makes the compilation finish, but then I get unresolved symbols error
> >>when attempting to run applications. Any ideas ?
> >>
> >>Another problem is that the ssh adaptor doesn't get compiled by default,
> >>and when manually trying to build it from it's directory, fails with 
> >>various
> >>errors related to not finding boost/process.hpp. I'm using boost-1.34.1,
> >>but don't see process.hpp existing on 1.37 either.
> >>
> >>Best,
> >>Katerina
> >
> >
> >
> 



-- 
Nothing is ever easy.