Message boards : Number crunching : Processing of chunks of data takes too long
Message board moderation

To post messages, you must log in.

AuthorMessage
Grzegorz Skoczylas

Send message
Joined: 25 Aug 20
Posts: 6
Credit: 5,852,467
RAC: 45
Message 4703 - Posted: 24 May 2021, 11:35:09 UTC
Last modified: 24 May 2021, 12:09:06 UTC

Processing a chunk of data often takes 4 days of CPU work. In reality, it takes much longer because data from another project is also processed in the menaantime.

In my opinion, these portions of data are too large. In the meantime, the computer load of other tasks may change, or other circumstances may arise (e.g. a few days off). As a result, I can waste, for example, 3.5 days of processor operation because, as a result of such circumstances, calculations may be completed well after the deadline.

I believe that processing portions of data should not take more than a dozen CPU hours, and ideally the computation can be completed in a few CPU hours. Otherwise, the larger the portions of data, the more CPU time can potentially be wasted.
ID: 4703 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Krzysztof Piszczek - wspieram ...
Project administrator
Project developer
Project tester
Avatar

Send message
Joined: 4 Feb 15
Posts: 841
Credit: 144,180,465
RAC: 2
Message 4704 - Posted: 24 May 2021, 11:50:51 UTC - in response to Message 4703.  
Last modified: 24 May 2021, 11:54:05 UTC

Zazwyczaj pojedyncze zadanie wykonuje się (w zależności od CPU i systemu operacyjnego)od 2 do maksymalnie 4 godzin.
Jeżeli u Ciebie wykonuje się powyżej 8 godzin i korzystasz z Linuksa - może to oznaczać jedną z rzadko spotykanych kombinacji parametrów wejściowych powodujących ekstremalnie długi czas liczenia. Sugeruję anulowanie takiego zadania.

Edit:
Widzę, że liczysz na dwóch komputerach, z których jeden ma czasu "normalne", natomiast drugi bardzo długie... Sugeruję przyjrzeć się tej maszynie, zwłaszcza pod kątem temperatur CPU...
Krzysztof 'krzyszp' Piszczek

Member of Radioactive@Home team
My Patreon profile
Universe@Home on YT
ID: 4704 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
tito

Send message
Joined: 21 Feb 15
Posts: 14
Credit: 862,506,967
RAC: 1,224
Message 4705 - Posted: 25 May 2021, 6:35:49 UTC - in response to Message 4703.  

Stop any other project and try Universe only.
Your Run time is ~ 60000sec while CPU time is ~ 10000sec
Anyway mixing different projects are usually bad idea. One main and one as back up with "zero" resources is optimal.

And try Ubuntu 20.04 :) (can be run in Windows 10)
ID: 4705 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Grzegorz Skoczylas

Send message
Joined: 25 Aug 20
Posts: 6
Credit: 5,852,467
RAC: 45
Message 4707 - Posted: 25 May 2021, 10:33:28 UTC - in response to Message 4704.  

Tak, liczę na dwóch komputerach. Oba z Windows 10.

    Na jednym z nich, dobre kilka lat starszym i słabszym, zadania Universe są liczone przeważnie około 20 godzin.
    Na drugim, nowszym i silniejszym, zadania Universe są czasem liczone kilkanaście godzin, ale często 2-4 dni.


Oprócz różnej mocy procesora, na starszym (desktop) jest tylko grafika Intel, a na nowszym (laptop) - również nVidia.

Nie znam szczegółów działania Boinc. Kiedyś zauważyłem, że czasem komputery dość znacznie różniły się mocą obliczeniową, a zadania były obliczane mniej więcej tak samo długo. W związku z tym nabrałem podejrzenia, że być może wielkość zadania jest dostosowywana do możliwości komputera. Dlatego, na wszelki wypadek, uruchomiłem test wydajności Boinc, ale tym razem nie zauważyłem jakiejś istotnej zmiany.

Kiedyś liczyłem również Seti@Home. Na innym komputerze (obecnie nieużywany) niektóre zadania strasznie długo się liczyły. Okazało się, że w tamtym projekcie zadania wykorzystujące do obliczeń kartę grafiki nie współpracowały dobrze z moim modelem karty graficznej Intel. W opcjach mojego profilu musiałem zaznaczyć, że nie chcę zdań wykorzystujących kartę grafiki Intel.

W Universe nie zaznaczyłem takiej opcji. Nie wiem, czy warto. Chyba nie ma to znaczenia, bo widzę, że zadania o takiej samej nawie (Universe BHSpin v2.0.19) czasem są obliczane w kilkanaście godzin, a czasem (częściej) w 2-4 dni.

ID: 4707 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Grzegorz Skoczylas

Send message
Joined: 25 Aug 20
Posts: 6
Credit: 5,852,467
RAC: 45
Message 4708 - Posted: 25 May 2021, 15:36:49 UTC - in response to Message 4705.  

It seems to me that counting several projects has almost nothing to do here. If I have several projects and I allocate the same amount of resources to each, it takes correspondingly longer to calculate each project in real time. Nevertheless, the amount of CPU time needed to compute a portion of the data is basically the same.

I can try installing BOINC in Ubuntu (on Windows).Is there a description somewhere on how to do it?

I use Throttle (https://efmer.com/download-tthrottle) to keep my CPU from overheating by BOINC tasks. Is there any similar program for Ubuntu?
ID: 4708 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
rsNeutrino

Send message
Joined: 1 Nov 17
Posts: 29
Credit: 291,940,933
RAC: 415
Message 4710 - Posted: 26 May 2021, 0:02:34 UTC - in response to Message 4708.  
Last modified: 26 May 2021, 0:10:42 UTC

Hi there!

Just in case the following is a bit too hard in English to understand, this is an extremely good translator that might help: https://www.deepl.com/translator

Here is a thread that might has useful info for running BOINC on Ubuntu inside Hyper-V on/beside Windows, which about triples Universe task performace in comparison to Windows: https://universeathome.pl/universe/forum_thread.php?id=550
In my case my main PC with a Ryzen 1700 is running since about April 2020 permanently, with 75% Universe and 25% Rosetta at the moment on 14/16 threads, overclocked to 3.6 GHz and undervolted to 1.20 V rock-stable. Last boot was 80 days ago...

I use Throttle (https://efmer.com/download-tthrottle) to keep my CPU from overheating by BOINC tasks. Is there any similar program for Ubuntu?

I'm not familiar with this tool, but I have a suspicion...
Have you done performance or points per day comparisions with and without this tool running? If your CPU gets too hot when used fully, tried using a lower fixed number of cores defined by the BOINC client? I see two computers in your profile, with Core i7 CPUs, so they have 2 threads per core I assume?
Each core can calculate 2 tasks at the same time, but one has to understand that this gets a lower additional performance in comparison to using one core for one task. In "Computing preferences" I would try "Use at most 49 % of the CPUs", to put one core to sleep and let the others run one task each, the OS manages this automatically, and "Use at most 100 % CPU time".
The cores sleeping will be switched around fast, so task manager will probably show about the same usage for each at below 50%

If your CPU gets too hot in general, I would suggest using solutions that go some levels deeper. First thing if possible would be cleaning the cooler or look for cooling solution improvements if it's a desktop.
A system should be able to run full blast with everything at the same time and be stable. If it's not stable or temps go above 85°C during load, something should be done. If above is not possible, I would look into lowering CPU clock speed and undervolting to reduce the amount of power flowing into the CPU, wich can work wonders, like 10% less peak performance for 50% less power consumption = heat. Dependent on the system there may be tools from the manufacturer to fine-tune it from Windows or inside the BIOS/UEFI. Maybe even Windows Power Management profiles could work if it changes the CPU clock speed.
IntelBurnTest is my got-to tool to test if a CPU is stable and its cooling enough, this thing hammers the temperature through the roof like nothing else and checks for calculation errors. The CPU should be tuned to NOT produce more heat while running this tool, than the cooler can remove. If temp is >85°C within 15 minutes the fan speed needs to be higher. if it can't get higher, CPU clock and/or voltage needs to be lower.
Higher voltage opens higher clock speed and increases stability but increases heat and it must stay under a certain critical level to not damage the CPU. Higher clock speed increases performance but also increases heat and decreases stability.
ID: 4710 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Grzegorz Skoczylas

Send message
Joined: 25 Aug 20
Posts: 6
Credit: 5,852,467
RAC: 45
Message 4712 - Posted: 26 May 2021, 9:19:27 UTC - in response to Message 4710.  

That is a very precise description. Thanks :-)

Throttle program mentioned by me in previous post is very convenient. You define maximum temperature for CPU and GPU, and the program automatically controls performance of BOINC processes not to exceed this temperature. This works very well if you are not using a dedicated computer for BOINC calculations. These computers of mine are used for different tasks. Sometimes they are more loaded, sometimes - less or almost not at all. I simply want to allocate free computer resources to BOINC calculations, but I don't want those computers to be noisy due to CPU overheating.
ID: 4712 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : Number crunching : Processing of chunks of data takes too long




Copyright © 2024 Copernicus Astronomical Centre of the Polish Academy of Sciences
Project server and website managed by Krzysztof 'krzyszp' Piszczek