Recommendations for scheduler

classic Classic list List threaded Threaded
52 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Recommendations for scheduler

Alan McKinnon-2
Hi,

Up-front disclaimer: Mostly [OT] post. But at least I'll test drive it
on Gentoo before putting it in production :-)

New job, new environment. Existing persons suffer from
5-year-old-with-a-hammer syndrome and assume cron is the solution to all
ills. Result: a towering edifice of cron jobs that may or may not
clobber each other's work, may or may not work at all, and implement no
error handling at all. But my god, can they spew out mail from STOUT


But cron has only one event trigger: wall-clock time. And it's a very
blunt weapon. I'm looking for recommendations of alternative schedulers
that satisfy real-world business needs that need some other event
trigger than what the time is right now.

For those familiar with it, I'm looking for something with the useful
feature set, without the useless features and without the price tag of
ControlM

Anyone care to share experiences?


--
Alan McKinnon
[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Recommendations for scheduler

Сергей
For example in crontab */3 means "every three hours/minutes/etc".

2014-08-01 21:32 GMT+04:00 Alan McKinnon <[hidden email]>:

> Hi,
>
> Up-front disclaimer: Mostly [OT] post. But at least I'll test drive it
> on Gentoo before putting it in production :-)
>
> New job, new environment. Existing persons suffer from
> 5-year-old-with-a-hammer syndrome and assume cron is the solution to all
> ills. Result: a towering edifice of cron jobs that may or may not
> clobber each other's work, may or may not work at all, and implement no
> error handling at all. But my god, can they spew out mail from STOUT
>
>
> But cron has only one event trigger: wall-clock time. And it's a very
> blunt weapon. I'm looking for recommendations of alternative schedulers
> that satisfy real-world business needs that need some other event
> trigger than what the time is right now.
>
> For those familiar with it, I'm looking for something with the useful
> feature set, without the useless features and without the price tag of
> ControlM
>
> Anyone care to share experiences?
>
>
> --
> Alan McKinnon
> [hidden email]
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Recommendations for scheduler

Сергей
Also you can have a look at anacron.

Reply | Threaded
Open this post in threaded view
|

Re: Recommendations for scheduler

James-2
In reply to this post by Alan McKinnon-2
Alan McKinnon <alan.mckinnon <at> gmail.com> writes:


> New job, new environment. Existing persons suffer from
> 5-year-old-with-a-hammer syndrome and assume cron is the solution to all
> ills. Result: a towering edifice of cron jobs that may or may not
> clobber each other's work, may or may not work at all, and implement no
> error handling at all. But my god, can they spew out mail from STOUT

Sounds like a department full of computer scientist I inherited a few
decades ago...........

I know nothing bout chronos, but I find it an interesting read....ymmv.


http://nerds.airbnb.com/introducing-chronos/
http://airbnb.github.io/chronos/
https://github.com/airbnb/chronos


cheers mate!

James




Reply | Threaded
Open this post in threaded view
|

Re: Recommendations for scheduler

Alan McKinnon-2
In reply to this post by Сергей
On 01/08/2014 19:50, Сергей wrote:
> Also you can have a look at anacron.
>
>
>


Unfortunately, anacron doesn't suit my needs at all. Here's how anacron
works:

this bunch of job will all happen today regardless of what time it is.
That's not what I need, I need something that has very little to do with
time. Example:

1. Start backup job on db server A
2. When complete, copy backup to server B and do a test import
3. If import succeeds, move backup to permanent storage and log the fact
4. If import fails, raise an alert and trigger the whole cycle to start
again at 1

Meanwhile,

1. All servers are regularly doing apt-get update and downloading .debs,
and applying security packages. Delay this on the db server if a backup
is in progress.

Meanwhile there is the regular Friday 5am code-publish cycle and
month-end finance runs - this is a DevOps environment.

Yes, I know I can hack something together with bash scripts and cron
with a truly insane number of flag files. But this doesn't work for sane
definitions of work involving other people. I can't expect my support
crew to read bash scripts they found from crontabs and figure out what
they mean. They need a picture that shows what will happen when and what
the environment looks like.

So basically I need something to replace bash and cron the same way
puppet replaces scp and for loops






--
Alan McKinnon
[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Recommendations for scheduler

Alan McKinnon-2
In reply to this post by James-2
On 01/08/2014 20:17, James wrote:

> Alan McKinnon <alan.mckinnon <at> gmail.com> writes:
>
>
>> New job, new environment. Existing persons suffer from
>> 5-year-old-with-a-hammer syndrome and assume cron is the solution to all
>> ills. Result: a towering edifice of cron jobs that may or may not
>> clobber each other's work, may or may not work at all, and implement no
>> error handling at all. But my god, can they spew out mail from STOUT
>
> Sounds like a department full of computer scientist I inherited a few
> decades ago...........

I've met folks like that....
Brilliant in their chosen field but completely useless outside it? The
kind of fellows who see nothing wrong with eating a barbeque'd steak
with a spoon because they can get a result?

>
> I know nothing bout chronos, but I find it an interesting read....ymmv.
>
>
> http://nerds.airbnb.com/introducing-chronos/
> http://airbnb.github.io/chronos/
> https://github.com/airbnb/chronos

Aaaaaaaah, now this sounds like something I can use. Proper dependency
chains, Restful JSON interface so the devs can write code to drive it in
automation.

Good find, thanks!




>
>
> cheers mate!
>
> James
>
>
>
>
>
>


--
Alan McKinnon
[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Recommendations for scheduler

John Covici
Alan McKinnon <[hidden email]> wrote:

> On 01/08/2014 20:17, James wrote:
> > Alan McKinnon <alan.mckinnon <at> gmail.com> writes:
> >
> >
> >> New job, new environment. Existing persons suffer from
> >> 5-year-old-with-a-hammer syndrome and assume cron is the solution to all
> >> ills. Result: a towering edifice of cron jobs that may or may not
> >> clobber each other's work, may or may not work at all, and implement no
> >> error handling at all. But my god, can they spew out mail from STOUT
> >
> > Sounds like a department full of computer scientist I inherited a few
> > decades ago...........
>
> I've met folks like that....
> Brilliant in their chosen field but completely useless outside it? The
> kind of fellows who see nothing wrong with eating a barbeque'd steak
> with a spoon because they can get a result?
>
> >
> > I know nothing bout chronos, but I find it an interesting read....ymmv.
> >
> >
> > http://nerds.airbnb.com/introducing-chronos/
> > http://airbnb.github.io/chronos/
> > https://github.com/airbnb/chronos
>
> Aaaaaaaah, now this sounds like something I can use. Proper dependency
> chains, Restful JSON interface so the devs can write code to drive it in
> automation.
>
> Good find, thanks!

Unless I am missing something, chronos is not in the tree at all.

--
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici
         [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Recommendations for scheduler

Martin Vaeth-2
In reply to this post by Alan McKinnon-2
Alan McKinnon <[hidden email]> wrote:
>
> But cron has only one event trigger: wall-clock time. And it's a very
> blunt weapon. I'm looking for recommendations of alternative schedulers
> that satisfy real-world business needs that need some other event
> trigger than what the time is right now.

I had a similar need recently, and since the discussion in

https://forums.gentoo.org/viewtopic-t-992780-highlight-.html

had led to nothing satisfactory for me, I have written a
scheduler tool which serves my needs
(which might very well differ from yours...):

The corresponding tool is still in beta testing phase:
https://github.com/vaeth/schedule/

You can install it from the mv overlay (available over layman).


Reply | Threaded
Open this post in threaded view
|

Re: Recommendations for scheduler

J. Roeleveld
In reply to this post by Alan McKinnon-2
On 1 August 2014 19:32:36 CEST, Alan McKinnon <[hidden email]> wrote:

>Hi,
>
>Up-front disclaimer: Mostly [OT] post. But at least I'll test drive it
>on Gentoo before putting it in production :-)
>
>New job, new environment. Existing persons suffer from
>5-year-old-with-a-hammer syndrome and assume cron is the solution to
>all
>ills. Result: a towering edifice of cron jobs that may or may not
>clobber each other's work, may or may not work at all, and implement no
>error handling at all. But my god, can they spew out mail from STOUT
>
>
>But cron has only one event trigger: wall-clock time. And it's a very
>blunt weapon. I'm looking for recommendations of alternative schedulers
>that satisfy real-world business needs that need some other event
>trigger than what the time is right now.
>
>For those familiar with it, I'm looking for something with the useful
>feature set, without the useless features and without the price tag of
>ControlM
>
>Anyone care to share experiences?

I'm also looking for a free alternative.
At most of my clients, I see Tivoli Workload Scheduler (TWS) being used a lot.

It has most things what you want from an intelligent multi host scheduler. Unfortunately,  it also comes with a corresponding price tag.

If anyone knows of an OS project with comparable features, please let me know.
Failing this, it is on my list to start writing one myself when I get some spare time.

--
Joost
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Reply | Threaded
Open this post in threaded view
|

Re: Recommendations for scheduler

J. Roeleveld
In reply to this post by James-2
On 1 August 2014 20:17:05 CEST, James <[hidden email]> wrote:

>Alan McKinnon <alan.mckinnon <at> gmail.com> writes:
>
>
>> New job, new environment. Existing persons suffer from
>> 5-year-old-with-a-hammer syndrome and assume cron is the solution to
>all
>> ills. Result: a towering edifice of cron jobs that may or may not
>> clobber each other's work, may or may not work at all, and implement
>no
>> error handling at all. But my god, can they spew out mail from STOUT
>
>Sounds like a department full of computer scientist I inherited a few
>decades ago...........
>
>I know nothing bout chronos, but I find it an interesting read....ymmv.
>
>
>http://nerds.airbnb.com/introducing-chronos/
>http://airbnb.github.io/chronos/
>https://github.com/airbnb/chronos
>
>
>cheers mate!
>
>James

Looks interesting.
Apart from it requiring a clustered environment (mesos).

Unless I misunderstand the part where it says it runs on top of mesos?

--
Joost
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Reply | Threaded
Open this post in threaded view
|

Re: Recommendations for scheduler

J. Roeleveld
In reply to this post by Martin Vaeth-2
On 1 August 2014 23:02:11 CEST, Martin Vaeth <[hidden email]> wrote:

>Alan McKinnon <[hidden email]> wrote:
>>
>> But cron has only one event trigger: wall-clock time. And it's a very
>> blunt weapon. I'm looking for recommendations of alternative
>schedulers
>> that satisfy real-world business needs that need some other event
>> trigger than what the time is right now.
>
>I had a similar need recently, and since the discussion in
>
>https://forums.gentoo.org/viewtopic-t-992780-highlight-.html
>
>had led to nothing satisfactory for me, I have written a
>scheduler tool which serves my needs
>(which might very well differ from yours...):
>
>The corresponding tool is still in beta testing phase:
>https://github.com/vaeth/schedule/
>
>You can install it from the mv overlay (available over layman).

Going to have a look at this soon.

What are the features it currently has already and what are you planning on adding?

--
Joost
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Reply | Threaded
Open this post in threaded view
|

Re: Recommendations for scheduler

Martin Vaeth-2
J. Roeleveld <[hidden email]> wrote:
>>https://github.com/vaeth/schedule/
>
> What are the features it currently has already

This is hard to answer, since at a first glance the whole thing
does not even look like a scheduler: It looks more like a means to
communicate with some server, but after the discussions in the
gentoo forums, it became clear to my surprise that this is all
what is needed for the use cases I had in mind:
The "real" scheduler driving the whole thing can be a tiny script
(in shell or any other language) which just communicates with
that server.

To understand whether this can solve your problems, it is
probably best if you look at the examples in the README
(and/or the mentioned discussion in the gentoo forum).

> and what are you planning on adding?

Since it is sufficient for my purposes, I am currently not
planning to add anything (except possibly bug fixes or if I run
into a problem which I cannot solve with it).
Patches for extensions are welcome, of course.
(Also suggestions without patches are welcome, but my time is
currently very limited, and I do not make any promises.)


Reply | Threaded
Open this post in threaded view
|

Re: Recommendations for scheduler

Alan McKinnon-2
In reply to this post by John Covici
On 01/08/2014 21:35, [hidden email] wrote:

> Alan McKinnon <[hidden email]> wrote:
>
>> On 01/08/2014 20:17, James wrote:
>>> Alan McKinnon <alan.mckinnon <at> gmail.com> writes:
>>>
>>>
>>>> New job, new environment. Existing persons suffer from
>>>> 5-year-old-with-a-hammer syndrome and assume cron is the solution to all
>>>> ills. Result: a towering edifice of cron jobs that may or may not
>>>> clobber each other's work, may or may not work at all, and implement no
>>>> error handling at all. But my god, can they spew out mail from STOUT
>>>
>>> Sounds like a department full of computer scientist I inherited a few
>>> decades ago...........
>>
>> I've met folks like that....
>> Brilliant in their chosen field but completely useless outside it? The
>> kind of fellows who see nothing wrong with eating a barbeque'd steak
>> with a spoon because they can get a result?
>>
>>>
>>> I know nothing bout chronos, but I find it an interesting read....ymmv.
>>>
>>>
>>> http://nerds.airbnb.com/introducing-chronos/
>>> http://airbnb.github.io/chronos/
>>> https://github.com/airbnb/chronos
>>
>> Aaaaaaaah, now this sounds like something I can use. Proper dependency
>> chains, Restful JSON interface so the devs can write code to drive it in
>> automation.
>>
>> Good find, thanks!
>
> Unless I am missing something, chronos is not in the tree at all.
>

Correct, it isn't in the tree. But there's nothing stopping me from
getting it in there

--
Alan McKinnon
[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Recommendations for scheduler

Alan McKinnon-2
In reply to this post by Martin Vaeth-2
On 01/08/2014 23:02, Martin Vaeth wrote:

> Alan McKinnon <[hidden email]> wrote:
>>
>> But cron has only one event trigger: wall-clock time. And it's a very
>> blunt weapon. I'm looking for recommendations of alternative schedulers
>> that satisfy real-world business needs that need some other event
>> trigger than what the time is right now.
>
> I had a similar need recently, and since the discussion in
>
> https://forums.gentoo.org/viewtopic-t-992780-highlight-.html

Interesting thread :-)

Conceptually, your needs are the same as mine - sequence defined by
something other than wall-clock time.
The responders there do the same thing as I experience - tunnel vision
with regard to cron. Sysadmins are used to cron and sadly most of us
want to ram a purely cron-based solution into places where it most
certainly does not belong.

Business rules very seldom fit easily into a cron model, they usually
rely on a defined sequence


>
> had led to nothing satisfactory for me, I have written a
> scheduler tool which serves my needs
> (which might very well differ from yours...):
>
> The corresponding tool is still in beta testing phase:
> https://github.com/vaeth/schedule/
>
> You can install it from the mv overlay (available over layman).

Nice, thanks for the link :-)

Now I have two projects to evaluate.


--
Alan McKinnon
[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Recommendations for scheduler

Alan McKinnon-2
In reply to this post by J. Roeleveld
On 01/08/2014 23:13, J. Roeleveld wrote:

> On 1 August 2014 19:32:36 CEST, Alan McKinnon <[hidden email]> wrote:
>> Hi,
>>
>> Up-front disclaimer: Mostly [OT] post. But at least I'll test drive it
>> on Gentoo before putting it in production :-)
>>
>> New job, new environment. Existing persons suffer from
>> 5-year-old-with-a-hammer syndrome and assume cron is the solution to
>> all
>> ills. Result: a towering edifice of cron jobs that may or may not
>> clobber each other's work, may or may not work at all, and implement no
>> error handling at all. But my god, can they spew out mail from STOUT
>>
>>
>> But cron has only one event trigger: wall-clock time. And it's a very
>> blunt weapon. I'm looking for recommendations of alternative schedulers
>> that satisfy real-world business needs that need some other event
>> trigger than what the time is right now.
>>
>> For those familiar with it, I'm looking for something with the useful
>> feature set, without the useless features and without the price tag of
>> ControlM
>>
>> Anyone care to share experiences?
>
> I'm also looking for a free alternative.
> At most of my clients, I see Tivoli Workload Scheduler (TWS) being used a lot.
>
> It has most things what you want from an intelligent multi host scheduler. Unfortunately,  it also comes with a corresponding price tag.

I have an unusual boss. He's a business owner and quite naturally
profit-driven. He also employs smart people and expects us to maintain
systems in-house.

He's also a zealous FLOSS fan.

So when I present him a price tag for software his first question is
always "is there any free as in freedom software suited for the job?"

I'm still trying to wrap my brains around dealing with a boss that
thinks like this :-)

>
> If anyone knows of an OS project with comparable features, please let me know.
> Failing this, it is on my list to start writing one myself when I get some spare time.
>
> --
> Joost
>


--
Alan McKinnon
[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Recommendations for scheduler

J. Roeleveld

On Saturday, August 02, 2014 11:33:30 AM Alan McKinnon wrote:

> On 01/08/2014 23:13, J. Roeleveld wrote:

> > On 1 August 2014 19:32:36 CEST, Alan McKinnon <[hidden email]> wrote:

> >> Hi,

> >>

> >> Up-front disclaimer: Mostly [OT] post. But at least I'll test drive it

> >> on Gentoo before putting it in production :-)

> >>

> >> New job, new environment. Existing persons suffer from

> >> 5-year-old-with-a-hammer syndrome and assume cron is the solution to

> >> all

> >> ills. Result: a towering edifice of cron jobs that may or may not

> >> clobber each other's work, may or may not work at all, and implement no

> >> error handling at all. But my god, can they spew out mail from STOUT

> >>

> >>

> >> But cron has only one event trigger: wall-clock time. And it's a very

> >> blunt weapon. I'm looking for recommendations of alternative schedulers

> >> that satisfy real-world business needs that need some other event

> >> trigger than what the time is right now.

> >>

> >> For those familiar with it, I'm looking for something with the useful

> >> feature set, without the useless features and without the price tag of

> >> ControlM

> >>

> >> Anyone care to share experiences?

> >

> > I'm also looking for a free alternative.

> > At most of my clients, I see Tivoli Workload Scheduler (TWS) being used a

> > lot.

> >

> > It has most things what you want from an intelligent multi host scheduler.

> > Unfortunately, it also comes with a corresponding price tag.

> I have an unusual boss. He's a business owner and quite naturally

> profit-driven. He also employs smart people and expects us to maintain

> systems in-house.

>

> He's also a zealous FLOSS fan.

>

> So when I present him a price tag for software his first question is

> always "is there any free as in freedom software suited for the job?"

 

Depends on the specific requirements.

If you want:

- time based start of a schedule

- dependencies in said schedules and between schedules which can delay the actual start

- stop of schedule if error occurs

- ability to restart schedule from crashed point

- have schedules operate over multiple machines (eg. part run on database, some on a compute-cluster, some other bit making nice graphs and printing it,...)

 

Then you might be out of luck.

If anyone has something that is already going along these lines, please let me know. I am more then willing to spend time and effort to assist in the development. Doing a project like that on my own in my extremely limited free time is not really an option.

 

> I'm still trying to wrap my brains around dealing with a boss that

> thinks like this :-)

 

Hehe :)

 

--

Joost

Reply | Threaded
Open this post in threaded view
|

Re: Recommendations for scheduler

J. Roeleveld
In reply to this post by Alan McKinnon-2

On Saturday, August 02, 2014 11:18:32 AM Alan McKinnon wrote:

> On 01/08/2014 21:35, [hidden email] wrote:

> > Alan McKinnon <[hidden email]> wrote:

> >> On 01/08/2014 20:17, James wrote:

> >>> Alan McKinnon <alan.mckinnon <at> gmail.com> writes:

> >>>> New job, new environment. Existing persons suffer from

> >>>> 5-year-old-with-a-hammer syndrome and assume cron is the solution to

> >>>> all

> >>>> ills. Result: a towering edifice of cron jobs that may or may not

> >>>> clobber each other's work, may or may not work at all, and implement no

> >>>> error handling at all. But my god, can they spew out mail from STOUT

> >>>

> >>> Sounds like a department full of computer scientist I inherited a few

> >>> decades ago...........

> >>

> >> I've met folks like that....

> >> Brilliant in their chosen field but completely useless outside it? The

> >> kind of fellows who see nothing wrong with eating a barbeque'd steak

> >> with a spoon because they can get a result?

> >>

> >>> I know nothing bout chronos, but I find it an interesting read....ymmv.

> >>>

> >>>

> >>> http://nerds.airbnb.com/introducing-chronos/

> >>> http://airbnb.github.io/chronos/

> >>> https://github.com/airbnb/chronos

> >>

> >> Aaaaaaaah, now this sounds like something I can use. Proper dependency

> >> chains, Restful JSON interface so the devs can write code to drive it in

> >> automation.

> >>

> >> Good find, thanks!

> >

> > Unless I am missing something, chronos is not in the tree at all.

>

> Correct, it isn't in the tree. But there's nothing stopping me from

> getting it in there

 

Neither are the dependencies.

 

If you get it to work, don't forget to create a nice howto documentation as from what I found online, the documentation is incomplete and out of date.

 

--

Joost

Reply | Threaded
Open this post in threaded view
|

Re: Recommendations for scheduler

Alan McKinnon-2
In reply to this post by J. Roeleveld
On 02/08/2014 15:31, J. Roeleveld wrote:

> Depends on the specific requirements.
>
> If you want:
>
> - time based start of a schedule
>
> - dependencies in said schedules and between schedules which can delay
> the actual start
>
> - stop of schedule if error occurs
>
> - ability to restart schedule from crashed point
>
> - have schedules operate over multiple machines (eg. part run on
> database, some on a compute-cluster, some other bit making nice graphs
> and printing it,...)
>
>  
>
> Then you might be out of luck.
>
> If anyone has something that is already going along these lines, please
> let me know. I am more then willing to spend time and effort to assist
> in the development. Doing a project like that on my own in my extremely
> limited free time is not really an option.
>


Well, we've found 2 projects that at least in part seek to achieve our
general goals - chronos and Martin's new project.

Why don't we both fool around with them for a bit and get a sense of
what it will take to add features etc? Then we can meet back here and
discuss.

Always better to build on an existing foundation

--
Alan McKinnon
[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Recommendations for scheduler

James-2
Alan McKinnon <alan.mckinnon <at> gmail.com> writes:


> Well, we've found 2 projects that at least in part seek to achieve our
> general goals - chronos and Martin's new project.
> Why don't we both fool around with them for a bit and get a sense of
> what it will take to add features etc? Then we can meet back here and
> discuss. Always better to build on an existing foundation

Mesos looks promising for a variety of (Apache) reasons. Some key
technologies folks may want google about that are related:

Quincy (fair schedular)
Chronos (scheduler)
Hadoop (scheduler)
HDFS (clusterd file system)
http://gpo.zugaina.org/sys-cluster/apache-hadoop-common

Zookeeper (Fault tolerance)
SPARK ( optimized for interative jobs where a datase is resued in many
parallel operations (advanced math/science and many other apps.)
https://spark.apache.org/

Dryad  Torque   Mpiche2 MPI
Globus tookit

mesos_tech_report.pdf

It looks as though Amazon, google, facebook and many others
large in the Cluster/Cloud arena are using Mesos......?

So let's all post what we find, particularly in overlays.

hth,
James


Reply | Threaded
Open this post in threaded view
|

Re: Recommendations for scheduler

J. Roeleveld
On Saturday 02 August 2014 16:53:26 James wrote:

> Alan McKinnon <alan.mckinnon <at> gmail.com> writes:
> > Well, we've found 2 projects that at least in part seek to achieve our
> > general goals - chronos and Martin's new project.
> > Why don't we both fool around with them for a bit and get a sense of
> > what it will take to add features etc? Then we can meet back here and
> > discuss. Always better to build on an existing foundation
>
> Mesos looks promising for a variety of (Apache) reasons. Some key
> technologies folks may want google about that are related:
>
> Quincy (fair schedular)
> Chronos (scheduler)
> Hadoop (scheduler)

Hadoop not a scheduler. It's a framework for a Big Data clustered database.

> HDFS (clusterd file system)

Unless it's changed recently, not suitable for anything else then Hadoop and
contains a single point of failure.

> http://gpo.zugaina.org/sys-cluster/apache-hadoop-common
>
> Zookeeper (Fault tolerance)
> SPARK ( optimized for interative jobs where a datase is resued in many
> parallel operations (advanced math/science and many other apps.)
> https://spark.apache.org/
>
> Dryad  Torque   Mpiche2 MPI
> Globus tookit
>
> mesos_tech_report.pdf
>
> It looks as though Amazon, google, facebook and many others
> large in the Cluster/Cloud arena are using Mesos......?
>
> So let's all post what we find, particularly in overlays.

Unless you are dealing with Big Data projects, like Google, Facebook, Amazon,
big banks,... you don't have much use for those projects.

Mesos looks like a nice project, just like Hadoop and related are also nice.
But for most people, they are as usefull as using Exalytics.

A scheduler should not have a large set of dependencies that you wouldn't use
otherwise. That makes Chronos a non-option to me.

Martin's project looks promising, but doesn't store the schedules internally.
For repeating schedules, like what Alan was describing, you need to put those
into scripts and start those from an existing cron.

Of the 2, I think improving Martin's project is the most likely option for me
as it doesn't have additional dependencies and seems to be easily implemented.

--
Joost

123