Comments on Valued Lessons: The manycore era is already upon us (and python is...

Shuttleworth is half-right. 2 and 3 aren't mutuall...

2009-05-10T14:57:00.000-07:00

Shuttleworth is half-right. 2 and 3 aren't mutually exclusive.

Threads are important when scaling vertically and specially in the cloud environment where you pay per cpu/mem it's invaluable.

Process are better when scaling horizontally, but that's a better solution for really big applications.

I think both are necessary and as long as there are tools that allow me to abstract the scaling strategy I'm fine.

Not to say this approach is the be-all, end-all fo...

2009-03-14T21:34:00.000-07:00

Not to say this approach is the be-all, end-all for multi-core work, but good ol' process spawning gets a lot of real work done, fairly easily, without changing the way I have to work.

Too bad Blogger is brain dead about pre tags, but for example here's 75% (approx) of each core being used; I think I needed to put more load on this web app to jam it up to 100% total:

CPU %user %nice %sys
all 60.44 0.00 13.24
0 62.35 0.00 13.53
1 55.88 0.00 13.53
2 62.35 0.00 11.76
3 60.82 0.00 14.62

What I like about this is I can get incredible - by the standards of just a few years ago - amount of processing done, with commodity hardware, without radically changing how I have to develop.

Again, I'm not suggesting there isn't room for improvement... and some folks will be forced to by their own needs and circumstances. But it isn't all bad that not *everyone* has to.

I just ran into this myself, I was working on a pr...

2008-12-16T10:50:00.000-08:00

I just ran into this myself, I was working on a project that is very cpu bound, but also very parallel. I could run 1000 threads if my system would not grind to a halt doing so.

But, no matter what I set the thread count to, python only uses 25%.

I will have to continue looking into options, but I may just have to run 4 copies of the program to allow me to access the full power of my system for now.

I agree, the whole multicore thing has been out fo...

2008-10-27T03:01:00.000-07:00

I agree, the whole multicore thing has been out for a while now and I'm yet to see anyone address it with much thought.

I may have to try using multiple processes. I had...

2008-10-15T08:52:00.000-07:00

I may have to try using multiple processes. I had seen that module before but didn't realize it was so complete. Thanks for the tip.

But can we all agree that's really a kludge? It sounds a lot like when Java people defend not having first-class functions by saying "just use an interface". Sure, it's possible. But it's ugly and no fun.

Mark Shuttleworth brought this up in his Keynote a...

2008-10-15T07:43:00.000-07:00

Mark Shuttleworth brought this up in his Keynote at Pycon UK and the reaction seemed to fall into 3 camps.

1. Who cares?
2. Just spawn out processes and use processing.
3. I want threads, stop telling me how to design my app, aka Processes are too heavyweight on Windows.

I guess there are people out there in camp 3 who need threads, not processes, and just can't live with the GIL. As Stan mentioned those in camp 2 have a solution, but it seems more like a workaround than a solution :(

I have a recollection of a patch being posted that seemed to work on multi-core machines, but ran slower on single core machines and so was scrapped. When single-core becomes the exception, it may be that it is revisited, but who knows?

It's worth taking a look at the "processing" modul...

2008-10-15T06:01:00.000-07:00

It's worth taking a look at the "processing" module, which was renamed the "multiprocessing" module and put into Python 2.6. It provides a concurrency model that avoids the GIL by forking and using pipes and shared memory. Sure, that's a little ugly under the hood, but (multi)processing wraps it up in a nice looking API:

multiprocessing documentation