Message boards : Number crunching : Increase queued work units
Message board moderation

To post messages, you must log in.

AuthorMessage
Todd1561

Send message
Joined: 19 May 21
Posts: 13
Credit: 483,254,000
RAC: 0
Message 4984 - Posted: 9 Dec 2021, 19:23:48 UTC
Last modified: 9 Dec 2021, 19:24:27 UTC

Is there a way to increase the number of tasks I can keep in my queue? I'd like to keep a few days worth so if there's an outage with the project servers I can keep crunching until they get it sorted out. Currently, with my 16 thread CPU it'll only let me keep about 350 tasks waiting, which I can complete in about 15 hours. I've adjusted
work_buf_min_days 
and
work_buf_additional_days 
but I just get a "This computer has reached a limit on tasks in progress" message from the server. The only way I've been able to pull more is by artificially boosting my
ncpus 
count but that has unintended consequences I don't like.
toddnelson.net: Astro pics, solar stats, free software
ID: 4984 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
mmonnin

Send message
Joined: 2 Jun 16
Posts: 169
Credit: 317,253,046
RAC: 6
Message 4985 - Posted: 9 Dec 2021, 22:42:07 UTC

Not with 1 client
ID: 4985 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Keith Myers
Avatar

Send message
Joined: 10 May 20
Posts: 308
Credit: 4,733,484,700
RAC: 14,480
Message 4987 - Posted: 9 Dec 2021, 23:22:25 UTC

The project limit is 24 tasks per thread. So your theoretical maximum is 384 tasks per day without the ncpu kludge.
You can look at this post which summarizes the limit. https://universeathome.pl/universe/forum_thread.php?id=606&postid=4968
The project only recognizes a maximum of 64 threads per host even with the ncpus parameter involved.

A proud member of the OFA (Old Farts Association)
ID: 4987 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Todd1561

Send message
Joined: 19 May 21
Posts: 13
Credit: 483,254,000
RAC: 0
Message 4988 - Posted: 10 Dec 2021, 13:03:32 UTC

Ok thanks. That seems to match what I’m seeing. Do you think there’s any interest on the project maintainers side to increase that number? I’m assuming it was formulated years ago before the high core/thread count CPUs we have today. If so is this the place to request it?
toddnelson.net: Astro pics, solar stats, free software
ID: 4988 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Krzysztof Piszczek - wspieram ...
Project administrator
Project developer
Project tester
Avatar

Send message
Joined: 4 Feb 15
Posts: 841
Credit: 144,180,465
RAC: 2
Message 4989 - Posted: 10 Dec 2021, 14:49:05 UTC - in response to Message 4988.  

I will check and possibly increase the limit over this weekend :)
Krzysztof 'krzyszp' Piszczek

Member of Radioactive@Home team
My Patreon profile
Universe@Home on YT
ID: 4989 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Todd1561

Send message
Joined: 19 May 21
Posts: 13
Credit: 483,254,000
RAC: 0
Message 4990 - Posted: 10 Dec 2021, 14:51:33 UTC - in response to Message 4989.  

Hey that's great, thanks!
toddnelson.net: Astro pics, solar stats, free software
ID: 4990 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Krzysztof Piszczek - wspieram ...
Project administrator
Project developer
Project tester
Avatar

Send message
Joined: 4 Feb 15
Posts: 841
Credit: 144,180,465
RAC: 2
Message 4991 - Posted: 10 Dec 2021, 17:52:59 UTC - in response to Message 4990.  

Limits changed, please confirm if it is enough.
Krzysztof 'krzyszp' Piszczek

Member of Radioactive@Home team
My Patreon profile
Universe@Home on YT
ID: 4991 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Todd1561

Send message
Joined: 19 May 21
Posts: 13
Credit: 483,254,000
RAC: 0
Message 4992 - Posted: 10 Dec 2021, 22:10:03 UTC
Last modified: 10 Dec 2021, 22:10:48 UTC

Thanks, looks like it bumped to 35 queued tasks per thread? Is that right? That helps but ideally I'd like to keep say 3 days worth in queue. That would be about 72 tasks per thread on my hardware.
toddnelson.net: Astro pics, solar stats, free software
ID: 4992 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
mmonnin

Send message
Joined: 2 Jun 16
Posts: 169
Credit: 317,253,046
RAC: 6
Message 4993 - Posted: 11 Dec 2021, 1:01:01 UTC

Set another project with zero priority if you're worried about running out of work. No one needs 3 days. If you run out of work with 1.5 days then so will everyone else.
ID: 4993 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Todd1561

Send message
Joined: 19 May 21
Posts: 13
Credit: 483,254,000
RAC: 0
Message 4994 - Posted: 11 Dec 2021, 1:40:27 UTC

It’s not about processors running idle, I already have a couple 0 weight backup projects. Nor is it about maintaining rank among the project participants. It’s about driving all my resources towards the project I’d like to support most. It’s in the best interest of the project, too. Because during an extended outage we could all keep working on Universe tasks rather than shift to a backup project.

Anyway, not looking to make a big thing about it. Just figured I’d pose the question. Thanks to Krzysztof for upping the limit!

Have a nice weekend everyone.
toddnelson.net: Astro pics, solar stats, free software
ID: 4994 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
mmonnin

Send message
Joined: 2 Jun 16
Posts: 169
Credit: 317,253,046
RAC: 6
Message 4995 - Posted: 12 Dec 2021, 18:35:39 UTC

Then run more clients.
Many projects have a lot lower task count per core even before U@H's change, esp for GPUs. Some have limits that only allow for 1-2 hours of work.
ID: 4995 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : Number crunching : Increase queued work units




Copyright © 2024 Copernicus Astronomical Centre of the Polish Academy of Sciences
Project server and website managed by Krzysztof 'krzyszp' Piszczek