Discussion:
Announcement: Alternate -j implementation
Michael Bishop
2012-04-19 11:36:37 UTC
Permalink
Hello Everyone,

I've recently finished an implementation of -j <max_concurrent_jobs> and would be delighted if the members of this list would take a detailed look at it

It's in this rake branch on github:

https://github.com/michaeljbishop/rake/commit/295c7a4d6d58b3e10c27b940a8259dd3e01c52f0
or
http://bit.ly/JMVARy

In addition, I've added a pull-request for the inclusion of the change to the master branch (allowing of course for further changes to match the style and simplicity of the original)

https://github.com/jimweirich/rake/pull/112

My apologies if this has been hashed out many times on this list. I'm hoping to offer a fresh look at the problem. Thank you for your consideration.

Sincerely,

Michael Bishop


---


SUMMARY
-------

USER INTERFACE:

The user-interface is simply a new -j flag that specifies the maximum number of tasks that can execute simultaneously (discussed here before as I've seen). If the -j flag is omitted, the old behavior of unlimited concurrent tasks is retained.


IMPLEMENTATION:

The implementation is inspired by Apple's Grand Central Dispatch model. The core of the problem is solved via changes to the file "multi-task.rb".

The current implementation of MultiTask#invoke_prerequisites spawns a new thread for each prerequisite and then waits to return until all the threads have completed.

In the alternate implementation, MultiTask#invoke_prerequisites creates a list of blocks inside which each prerequisite is called. Each block is then added to a thread-safe Queue for processing. A class-member ThreadPool is then expanded to include enough threads to consume and call the blocks, but only up to the limit as passed to -j.

Then, rather than MultiTask#invoke_prerequisites sleeping by joining to the newly spawned threads, it participates in the processing of the queued blocks while it waits. This prevents deadlock situations should more threads call MultiTask#invoke_prerequisites.

How then does MultiTask#invoke_prerequisites know when its prerequisites have finished? Before enqueing the prerequisite blocks, it surrounds the original prerequisite block with *another* block which maintains bookkeeping tasks as to when the prerequisite is completed and which thread is working on it. That block is what is added to the queue.

Two conditions need to be met for MultiTask#invoke_prerequisites to return:

1 - It notices that its prerequisites have all been processed
2 - It notices there are no more blocks on the queue but its prerequisites are not yet finished. In this case, it joins those threads that are still executing its prerequisites.

What is attractive to me about this implementation is that the flow retains the simplicity of the original: MultiTask#invoke_prerequisites sends all the prerequisites to be executed then waits until they are done.



THE CASE FOR -J
---------------

(Quoted from the pull-request)

PROBLEM SUMMARY:

Rake can be unusable for builds invoking large numbers of concurrent external processes.

PROBLEM DESCRIPTION:

Rake makes it easy to maximize concurrency in builds with its "multitask" function. When using rake to build non-ruby projects quite often rake needs to execute shell tasks to process files. Unfortunately, when executing multitasks, rake spawns a new thread for each task-prerequisite. This shouldn't cause problems when the build code is pure ruby (for green threads), but when the tasks are executing external processes, the sheer number of spawned processes can cause the machine to thrash. Additionally ruby can reach the maximum number of open files (presumably because it's reading stdout for all those processes).

SOLUTION SUMMARY:

This request includes the code to add support for a "--jobs NUMBER (-j)" command-line option to specify the number of simultaneous tasks to execute.

SOLUTION:

The solution creates a work queue to which blocks calling the task-prerequisites are added and a thread pool to process them. To prevent deadlock, the task that added the pre-requisites processes items on the queue (alongside the thread pool) until its prerequisites have been processed.

To maintain backward compatibility, not passing -j reverts to the old behavior of unlimited concurrent tasks.

REQUIREMENTS:

The Ruby version requirements remain the same. "multi-task.rb" adds two new requirements: 'thread' and 'set'
Hongli Lai
2012-04-19 14:34:08 UTC
Permalink
Have you also looked at http://drake.rubyforge.org? How does your
implementation differ from his?
Post by Michael Bishop
Hello Everyone,
I've recently finished an implementation of -j <max_concurrent_jobs> and
would be delighted if the members of this list would take a detailed look at
it
 https://github.com/michaeljbishop/rake/commit/295c7a4d6d58b3e10c27b940a8259dd3e01c52f0
 or
   http://bit.ly/JMVARy
In addition, I've added a pull-request for the inclusion of the change to
the master branch (allowing of course for further changes to match the style
and simplicity of the original)
 https://github.com/jimweirich/rake/pull/112
My apologies if this has been hashed out many times on this list. I'm hoping
to offer a fresh look at the problem. Thank you for your consideration.
Sincerely,
Michael Bishop
---
SUMMARY
-------
The user-interface is simply a new -j flag that specifies the maximum number
of tasks that can execute simultaneously (discussed here before as I've
seen). If the -j flag is omitted, the old behavior of unlimited concurrent
tasks is retained.
The implementation is inspired by Apple's Grand Central Dispatch model. The
core of the problem is solved via changes to the file "multi-task.rb".
The current implementation of MultiTask#invoke_prerequisites spawns a new
thread for each prerequisite and then waits to return until all the threads
have completed.
In the alternate implementation, MultiTask#invoke_prerequisites creates a
list of blocks inside which each prerequisite is called. Each block is then
added to a thread-safe Queue for processing. A class-member ThreadPool is
then expanded to include enough threads to consume and call the blocks, but
only up to the limit as passed to -j.
Then, rather than MultiTask#invoke_prerequisites sleeping by joining to the
newly spawned threads, it participates in the processing of the queued
blocks while it waits. This prevents deadlock situations should more threads
call MultiTask#invoke_prerequisites.
How then does MultiTask#invoke_prerequisites know when its prerequisites
have finished? Before enqueing the prerequisite blocks, it surrounds the
original prerequisite block with *another* block which maintains bookkeeping
tasks as to when the prerequisite is completed and which thread is working
on it. That block is what is added to the queue.
  1 - It notices that its prerequisites have all been processed
  2 - It notices there are no more blocks on the queue but its prerequisites
are not yet finished. In this case, it joins those threads that are still
executing its prerequisites.
What is attractive to me about this implementation is that the flow retains
the simplicity of the original: MultiTask#invoke_prerequisites sends all the
prerequisites to be executed then waits until they are done.
THE CASE FOR -J
---------------
(Quoted from the pull-request)
Rake can be unusable for builds invoking large numbers of concurrent external processes.
Rake makes it easy to maximize concurrency in builds with its "multitask"
function. When using rake to build non-ruby projects quite often rake needs
to execute shell tasks to process files. Unfortunately, when executing
multitasks, rake spawns a new thread for each task-prerequisite. This
shouldn't cause problems when the build code is pure ruby (for green
threads), but when the tasks are executing external processes, the sheer
number of spawned processes can cause the machine to thrash. Additionally
ruby can reach the maximum number of open files (presumably because it's
reading stdout for all those processes).
This request includes the code to add support for a "--jobs NUMBER (-j)"
command-line option to specify the number of simultaneous tasks to execute.
The solution creates a work queue to which blocks calling the
task-prerequisites are added and a thread pool to process them. To prevent
deadlock, the task that added the pre-requisites processes items on the
queue (alongside the thread pool) until its prerequisites have been
processed.
To maintain backward compatibility, not passing -j reverts to the old
behavior of unlimited concurrent tasks.
The Ruby version requirements remain the same. "multi-task.rb" adds two new
requirements: 'thread' and 'set'
_______________________________________________
Rake-devel mailing list
http://rubyforge.org/mailman/listinfo/rake-devel
--
Phusion | Ruby & Rails deployment, scaling and tuning solutions

Web: http://www.phusion.nl/
E-mail: ***@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)
Michael Bishop
2012-04-19 15:48:03 UTC
Permalink
Hi Hongli Lai,

This is a good question. I only knew of the Drake project as of last night so I haven't had a long period of time to investigate its implementation.

My initial analysis based on the Drake official pull-request announcement indicates a few differences but please add corrections if I am misinformed. I might be.

Drake links:

https://github.com/quix/rake/tree/mainline-merge-candidate#readme

http://quix.github.com/rake/files/doc/parallel_rdoc.html


For the purposes of discussion, I'll refer to my submitted rake as "mjbrake" here.


Dependency differences:
-----------------------

Drake relies on an external gem named comp_tree for parallelism. mjbrake relies on no library but instead, like the original rake, leverages the stack to make sure all the prerequisites are executed in order.


Code size differences:
----------------------

Github shows the change for mjbrake at

"Showing 3 changed files with 74 additions and 7 deletions."

Drake shows 79 lines (+257 lines of code for comp_tree). It's quite possible this is due to drake having a larger feature set. mjbrake is focused on changing multitask to limit the total number of concurrent tasks.


Implementation differences:
---------------------------

It's unclear to me, but it seems like Drake doesn't ask the rakefile author to specify what can be parallelized and instead, asks that the dependencies be correct and Drake will figure that out dynamically.

If this is true, I suspect the transition to a parallelized build might be more gentle to the user if they can build a rakefile starting with 'task' and migrate to 'multitask' as in the original rake. On the other hand, there is something elegant about not having to specify 'multitask' at all.

_ michael
Post by Hongli Lai
Have you also looked at http://drake.rubyforge.org? How does your
implementation differ from his?
Post by Michael Bishop
Hello Everyone,
I've recently finished an implementation of -j <max_concurrent_jobs> and
would be delighted if the members of this list would take a detailed look at
it
https://github.com/michaeljbishop/rake/commit/295c7a4d6d58b3e10c27b940a8259dd3e01c52f0
or
http://bit.ly/JMVARy
In addition, I've added a pull-request for the inclusion of the change to
the master branch (allowing of course for further changes to match the style
and simplicity of the original)
https://github.com/jimweirich/rake/pull/112
My apologies if this has been hashed out many times on this list. I'm hoping
to offer a fresh look at the problem. Thank you for your consideration.
Sincerely,
Michael Bishop
---
SUMMARY
-------
The user-interface is simply a new -j flag that specifies the maximum number
of tasks that can execute simultaneously (discussed here before as I've
seen). If the -j flag is omitted, the old behavior of unlimited concurrent
tasks is retained.
The implementation is inspired by Apple's Grand Central Dispatch model. The
core of the problem is solved via changes to the file "multi-task.rb".
The current implementation of MultiTask#invoke_prerequisites spawns a new
thread for each prerequisite and then waits to return until all the threads
have completed.
In the alternate implementation, MultiTask#invoke_prerequisites creates a
list of blocks inside which each prerequisite is called. Each block is then
added to a thread-safe Queue for processing. A class-member ThreadPool is
then expanded to include enough threads to consume and call the blocks, but
only up to the limit as passed to -j.
Then, rather than MultiTask#invoke_prerequisites sleeping by joining to the
newly spawned threads, it participates in the processing of the queued
blocks while it waits. This prevents deadlock situations should more threads
call MultiTask#invoke_prerequisites.
How then does MultiTask#invoke_prerequisites know when its prerequisites
have finished? Before enqueing the prerequisite blocks, it surrounds the
original prerequisite block with *another* block which maintains bookkeeping
tasks as to when the prerequisite is completed and which thread is working
on it. That block is what is added to the queue.
1 - It notices that its prerequisites have all been processed
2 - It notices there are no more blocks on the queue but its prerequisites
are not yet finished. In this case, it joins those threads that are still
executing its prerequisites.
What is attractive to me about this implementation is that the flow retains
the simplicity of the original: MultiTask#invoke_prerequisites sends all the
prerequisites to be executed then waits until they are done.
THE CASE FOR -J
---------------
(Quoted from the pull-request)
Rake can be unusable for builds invoking large numbers of concurrent external processes.
Rake makes it easy to maximize concurrency in builds with its "multitask"
function. When using rake to build non-ruby projects quite often rake needs
to execute shell tasks to process files. Unfortunately, when executing
multitasks, rake spawns a new thread for each task-prerequisite. This
shouldn't cause problems when the build code is pure ruby (for green
threads), but when the tasks are executing external processes, the sheer
number of spawned processes can cause the machine to thrash. Additionally
ruby can reach the maximum number of open files (presumably because it's
reading stdout for all those processes).
This request includes the code to add support for a "--jobs NUMBER (-j)"
command-line option to specify the number of simultaneous tasks to execute.
The solution creates a work queue to which blocks calling the
task-prerequisites are added and a thread pool to process them. To prevent
deadlock, the task that added the pre-requisites processes items on the
queue (alongside the thread pool) until its prerequisites have been
processed.
To maintain backward compatibility, not passing -j reverts to the old
behavior of unlimited concurrent tasks.
The Ruby version requirements remain the same. "multi-task.rb" adds two new
requirements: 'thread' and 'set'
_______________________________________________
Rake-devel mailing list
http://rubyforge.org/mailman/listinfo/rake-devel
--
Phusion | Ruby & Rails deployment, scaling and tuning solutions
Web: http://www.phusion.nl/
Chamber of commerce no: 08173483 (The Netherlands)
_______________________________________________
Rake-devel mailing list
http://rubyforge.org/mailman/listinfo/rake-devel
Jim Weirich
2012-04-19 15:48:04 UTC
Permalink
Post by Hongli Lai
Have you also looked at http://drake.rubyforge.org? How does your
implementation differ from his?
Michael can correct me if I'm wrong, but the -j option just limits the number of concurrent tasks used to service the "multitask" task targets. Multitask is an existing rake facility that the -j option makes more usable.

Drake actually changes the semantics of the regular rake tasks so that they run in parallel.
--
-- Jim Weirich
-- ***@gmail.com
Jos Backus
2012-04-19 16:21:54 UTC
Permalink
To be honest, I prefer the drake semantics as they are closer to make's.

I would love to see rake and drake merged.

Jos
--
Jos Backus
jos at catnook.com
Mark Watson
2012-04-19 17:03:21 UTC
Permalink
Post by Jos Backus
To be honest, I prefer the drake semantics as they are closer to make's.
Second that. The option to specify parallel builds for any task is
handy because the task may not derive from multitask, e.g. it could be
a file task to build a c library from multiple source files that you
want to build in parallel.

Personally I've used my own patch to rake to do this, that was before
drake, but I still use it due to drake having an extra dependency.

https://github.com/watsonmw/rakecpp/blob/master/minusj/minusj.rb
Post by Jos Backus
To be honest, I prefer the drake semantics as they are closer to make's.
I would love to see rake and drake merged.
Jos
--
Jos Backus
jos at catnook.com
_______________________________________________
Rake-devel mailing list
http://rubyforge.org/mailman/listinfo/rake-devel
Hongli Lai
2012-04-19 21:40:18 UTC
Permalink
Post by Mark Watson
Post by Jos Backus
To be honest, I prefer the drake semantics as they are closer to make's.
Second that. The option to specify parallel builds for any task is
handy because the task may not derive from multitask, e.g. it could be
a file task to build a c library from multiple source files that you
want to build in parallel.
Personally I've used my own patch to rake to do this, that was before
drake, but I still use it due to drake having an extra dependency.
https://github.com/watsonmw/rakecpp/blob/master/minusj/minusj.rb
I agree. I would really love to see a 'make -j' equivalent in Rake.
--
Phusion | Ruby & Rails deployment, scaling and tuning solutions

Web: http://www.phusion.nl/
E-mail: ***@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)
Steven Parkes
2012-04-20 02:25:38 UTC
Permalink
Post by Hongli Lai
Post by Mark Watson
Post by Jos Backus
To be honest, I prefer the drake semantics as they are closer to make's.
Second that. The option to specify parallel builds for any task is
handy because the task may not derive from multitask, e.g. it could be
a file task to build a c library from multiple source files that you
want to build in parallel.
Personally I've used my own patch to rake to do this, that was before
drake, but I still use it due to drake having an extra dependency.
https://github.com/watsonmw/rakecpp/blob/master/minusj/minusj.rb
I agree. I would really love to see a 'make -j' equivalent in Rake.
I went through this just this week, after getting tired of some C++ builds using only 1 core. I first looked at multitask but got stopped by the file dependence issue. Then I did some wider searching and stumbled across drake. Been happy ever since.

I used make a lot in the past and always made the effort to make all the dependencies explicit (rather than implicit through dependence list order) so that make -j would work. But a lot of Makefiles don't do this and break with make -j just as many rakefiles would.

I can see this being a bigger issue with rake since its expressiveness makes it a lot easier to write things that have side-effects and won't parallelize. (Just like I see people write rakefiles that don't converge: that rebuild artifacts because they haven't described the dependencies correctly.)

But in any case, for my part, with (d)rake -j, I don't go back to make anymore at all, which makes my world a little nicer.
Michael Bishop
2012-04-19 16:36:32 UTC
Permalink
Post by Jim Weirich
Post by Hongli Lai
Have you also looked at http://drake.rubyforge.org? How does your
implementation differ from his?
Michael can correct me if I'm wrong, but the -j option just limits the number of concurrent tasks used to service the "multitask" task targets. Multitask is an existing rake facility that the -j option makes more usable.
That is exactly right. It only affects Multitask.

Some additional interesting tidbits: There is a queue of blocks that is a class member of Multitask which all Multitask instances use to queue their prerequisites. So, your thread's call to MultiTask#invoke_prerequisites may end up processing prerequisites for another's call to MultiTask#invoke_prerequisites. No matter though, your call won't return until all your prerequisites are completed.
Post by Jim Weirich
Drake actually changes the semantics of the regular rake tasks so that they run in parallel.
Thank you for that clarification. I'd only assumed that, given my rather cursory look, and it's nice to have it confirmed.

_ michael
Michael Bishop
2012-04-19 16:28:24 UTC
Permalink
Post by Jim Weirich
Post by Hongli Lai
Have you also looked at http://drake.rubyforge.org? How does your
implementation differ from his?
Michael can correct me if I'm wrong, but the -j option just limits the number of concurrent tasks used to service the "multitask" task targets. Multitask is an existing rake facility that the -j option makes more usable.
That is exactly right. It only affects Multitask.

Some additional interesting tidbits: There is a queue of blocks that is a class member of Multitask which all Multitask instances use to queue their prerequisites. So, your thread's call to MultiTask#invoke_prerequisites may end up processing prerequisites for another's call to MultiTask#invoke_prerequisites. No matter though, your call won't return until all your prerequisites are completed.
Post by Jim Weirich
Drake actually changes the semantics of the regular rake tasks so that they run in parallel.
Thank you for that clarification. I'd only assumed that, given my rather cursory look, and it's nice to have it confirmed.

_ michael
Michael Bishop
2012-04-20 08:51:17 UTC
Permalink
Quick question for the list:

After reading more about drake, I wanted to summarize the differences for my own understanding.

Differences in drake from rake:

1 - All tasks are implicitly multitask
2 - There is a cap on the number of concurrent tasks
3 - There is a control to create a repeatable single-threaded random invocation of tasks (--randomize[=SEED])

Is this correct?

_ michael
Mark Watson
2012-04-20 21:33:40 UTC
Permalink
Post by Michael Bishop
1 - All tasks are implicitly multitask
Yes, in the sense that all tasks can be executed in parallel with other tasks.

Upside parallelism is octagonal to the method
('file'/'directory'/'task') used to define the task.

It has the downside that some tasks may not be suitable for executing
in parallel. E.g. the dependencies are wrong, or the task itself
incorrectly shares some resource with other tasks (say multiple tasks
modify the same file as an intermediate step during the task
execution). IMHO, making stuff work in parallel requires thinking
about the problem anyway and often leads to better code. If one
needed to make something single thread a mutex should do the trick.
So it's not that hard to be a good -j citizen :)
Post by Michael Bishop
2 - There is a cap on the number of concurrent tasks
Yes, this is whatever is set in the -j option.
Post by Michael Bishop
3 - There is a control to create a repeatable single-threaded random invocation of tasks (--randomize[=SEED])
Yes, it's to help people debug rakefile dependencies, as you've
probably figured :)
Post by Michael Bishop
After reading more about drake, I wanted to summarize the differences for my own understanding.
1 - All tasks are implicitly multitask
2 - There is a cap on the number of concurrent tasks
3 - There is a control to create a repeatable single-threaded random invocation of tasks (--randomize[=SEED])
Is this correct?
_ michael
_______________________________________________
Rake-devel mailing list
http://rubyforge.org/mailman/listinfo/rake-devel
Loading...