Message boards :
Number crunching :
Double your task throughput on Linux
Message board moderation
Previous · 1 · 2 · 3 · 4 · 5 · Next
Author | Message |
---|---|
Send message Joined: 10 May 20 Posts: 310 Credit: 4,733,484,700 RAC: 0 |
Baby steps Chooka. Up the learning curve. You might want to bookmark and visit a couple of beginner Linux websites for easy tips 'n tricks articles. Also a good library of past articles. https://itsfoss.com/ https://www.omgubuntu.co.uk/ A proud member of the OFA (Old Farts Association) |
Send message Joined: 30 Jun 16 Posts: 42 Credit: 309,815,029 RAC: 0 |
I ran a few task using the VM and Ubuntu. Times aren't as low as yours Keith but they went from 8000 - 9000 sec with Win 10 down to a more consistent 5300 sec with VM /Ubuntu. An interesting exercise. |
Send message Joined: 15 Aug 20 Posts: 37 Credit: 8,554,094,169 RAC: 0 |
you get some overhead with a virtualized environment so it'll never be as fast as bare metal. |
Send message Joined: 30 Jun 16 Posts: 42 Credit: 309,815,029 RAC: 0 |
Hi Ian&Steve, Yes Keith warned me of that. Gee that's some great equipment you guys have there. I regularly check the stats and some of you guys (especially TSBT) as slaughtering a few projects at the moment! My pockets aren't deep enough to keep up with all that hardware lol. It will be a big task for me to change BIOS settings and install a VM/Ubuntu across all 6 of my crunching pc's. There's some real merit it fewer machines with a ton of cores. Especially regarding Windows updates too. Urgh....good old Windows. Have a good weekend. |
Send message Joined: 10 May 20 Posts: 310 Credit: 4,733,484,700 RAC: 0 |
I ran a few task using the VM and Ubuntu. Times aren't as low as yours Keith but they went from 8000 - 9000 sec with Win 10 down to a more consistent 5300 sec with VM /Ubuntu. Yeah, I saw that. About 45 minutes faster. I don't know whether there are parameters you can set in the VM that would speed things up or not. Also different hypervisors might have lower overhead and can run tasks closer to bare metal. A proud member of the OFA (Old Farts Association) |
Send message Joined: 1 Nov 17 Posts: 29 Credit: 291,940,933 RAC: 0 |
If you open Task Manager and go into the Performance / CPU tab, on the lower right is shown if "Virtualization" is enabled or disabled. Image: https://www.intel.com/content/dam/support/us/en/images/processors/5486_image5.png If it is enabled, you are ready to use Hyper-V which runs as bare metal as it can get, practically no performace lost, because it is a hardware-accelerated Type I Hypervisor: Even your Window 10 itself kind of becomes a VM then (like OS1 in the image), all systems on the computer get equal and full core performance, while the root/host OS (Window 10 in this case) retains the "admin rights". If virtualization is not enabled, you have to go into the BIOS to enable it first. To enable and use Hyper-V after virtualization is enabled, this may help: https://techcommunity.microsoft.com/t5/itops-talk-blog/step-by-step-enabling-hyper-v-for-use-on-windows-10/ba-p/267945 Warning: As far as I know, Virtualbox is not able to run with Hyper-V enabled. (Because it doesn't like to run inside a VM itself) |
Send message Joined: 10 May 20 Posts: 310 Credit: 4,733,484,700 RAC: 0 |
So I think you are saying the Windows native Hyper-V hypervisor is a closer to bare metal solution compared to BOINC's Virtual-Box hypervisor? A proud member of the OFA (Old Farts Association) |
Send message Joined: 1 Nov 17 Posts: 29 Credit: 291,940,933 RAC: 0 |
Yes, Virtualbox is Type II, which runs on top of Windows, and may has to do some emulation, patching and hacks to run the code inside it, but It uses the virtualization extentions where it can. Hyper-V is Type I, which runs as its own mini-OS directly on the hardware. Windows runs on top of that, as does every other VM, with close to zero performance impact. VMs run parallel to Windows, not on top of Windows anymore, so all the code still runs natively on the CPU, as far as I'm aware. The linked article in my last post has commands to change the Windows boot mode to toggle the Hypervisor on and off. Installing/Uninstalling Hyper-V through Windows Features should do the same thing. I did some performance tests back in May and Ubuntu 20.04 on Virtualbox managed to run Universe task series 310 with a computation time of 40 to 50 Minutes. So not bad at all! (Ryzen 7 1700 @ 3.6 GHz) But the thing that made me switch from Virtualbox to Hyper-V was another no-go problem: I have 16 threads available and want to use 14 for BOINC, which works fine when using BOINC on Windows. But when running a Virtualbox VM with such a thread count, the whole PC begins to lag HARD, linke I struggled to get even the mouse moving on the host! In several forums people wrote that using many threads is a specific weakness for Virtualbox. Maybe running serveral VMs in parallel with less theads each could alleviate that, but there is overhead by having multiple BOINC instances, and multiple Linux OS to manage. Hyper-V to the rescue! Windows 10 Home users are out of luck, sadly, they can NOT use it, it's only a feature for Professional (and up). Those who have Windows 10 Pro just need to have virtualization enabled in BIOS, tick a checkmark at "Hyper-V" in the Windows Features (old system control panel) and are ready to use it. My PC runs practically non-stop since March, with around 90% CPU usage for Universe and Rosetta inside the Ubuntu VM, and at certain times Folding@home on Windows added to the mix. Just a restart e.g. for updates every few weeks, it is my daily driver and fully usable without noticable slowdown, except in certain games. But Hyper-V has a nice Pause button for that. ^^ Another thing: Hyper-V is a bit worse than Virtualbox concerning file and clipboard transfer which does not work between Windows and Ubuntu, and display resolution, which is quite low and not changable using normal options. The first problem can be circumvented by using AnyDesk, the second is a bit more complex, where a Linux boot option has to be added in a file to define a different resolution (mine is at 1600x900 now): https://virtualizationreview.com/Blogs/Virtual-Insider/2014/09/change-ubuntu-resolution-on-hyper-v-vm.aspx |
Send message Joined: 10 May 20 Posts: 310 Credit: 4,733,484,700 RAC: 0 |
Thanks for the mini-lesson in VM's. Total VM noob here. I've never needed to run any project that required BOINC's Virtual-Box image. Bare metal is good enough for me. A proud member of the OFA (Old Farts Association) |
Send message Joined: 30 Jun 16 Posts: 42 Credit: 309,815,029 RAC: 0 |
I'm a bit concerned that might be too much for me rsNeutrino. I've struggled just to navigate Linux lol. Virtualisation is enabled on my machine and I can see how to turn on Hyper V but I don't fully understand what will happen. Also, if VM won't work inside Hyper-V, how do I run Linux which is installed on the VM? |
Send message Joined: 1 Nov 17 Posts: 29 Credit: 291,940,933 RAC: 0 |
This might help: https://www.youtube.com/watch?v=A1NxPJ-PMH4 "Also, if VM won't work inside Hyper-V, how do I run Linux which is installed on the VM?" Do you mean the Ubuntu VM from Virtualbox? As long as Hyper-V is installed, it will refuse to work. If you run into problems, you can uninstall Hyper-V anytime the same way you installed it to get Virtualbox working again. There is a way to convert the virtual hard drive of the Virtualbox VM to the format Microsoft uses (vhdx files), but it's complicated... The most simple way is to use "Quick Create..." like in the video, select the latest Ubuntu version and install BOINC through the Ubuntu Software Center. Baby steps... and Google helps. You don't have to do anything if you don't know what is happening or what things are or do. No stress. Maybe take one unknown thing, when you have the time, and read a bit about it (maybe wikipedia) or look on youtube or reddit. 9 months ago I didn't know anything about Hyper-V, but the ability to crawl the internet gives much knowledge. Then experimenting within the limits of your knowledge to gain more at the edges where answers cannot be found so easily makes an expert. Another thing: the virtual HDD the quick create wizard creates is quite small (12 GB with 6 GB in use), but good enough for universe@home. Other projects may want more disk space, so that could lead to problems down the road, but I have just tested a way to expand the space later: Shutdown VM if it isn't already VM options > checkpoints > disable checkpoints (snapshots) select VM > lower panel: snapshots > delete whole snapshot tree (snapshot files will get merged into main vhdx file automatically) "Edit Disk..." on the right panel > find vhdx-file > expand > e.g. 30GB Start VM > install and launch GParted > it may ask to repair GPT to use the new space (do it) > expand last partition (the big one) to fill the added space If you want to learn more about the Linux filesystem, this might be insightful: https://www.youtube.com/watch?v=HbgzrKJvDRw |
Send message Joined: 30 Jun 16 Posts: 42 Credit: 309,815,029 RAC: 0 |
Thanks for the tips. |
Send message Joined: 1 Nov 17 Posts: 29 Credit: 291,940,933 RAC: 0 |
Great to hear that it worked that way for you! I dug around a bit and found other instructions that seem even more straightforward: https://docs.microsoft.com/en-us/answers/questions/29175/installation-of-hyper-v-on-windows-10-home.html https://github.com/MicrosoftDocs/Virtualization-Documentation/issues/915 https://pastebin.com/cX6nupy4 No idea if it still works. It seems to use the integrated DISM tool to download the missing Hyper-V packages into the local package repository, makes them available for activation under "Programs and Features" and Windows 10 Home does not block it. |
Send message Joined: 26 Nov 16 Posts: 11 Credit: 12,880,500 RAC: 0 |
Thanks rsNeutrino for detailed walkthrough. I want to try it but first I have to convert my Virtual box VMs to Hyper-v machines. And need to find a temporary HDD for trying and converting HDD images from vdi to vhd type. Lets see in a couple of days do I manage or not :) Seti@home Classic account User ID 955 member since 8 Sep 1999 classic CPU time 539,770 hours |
Send message Joined: 26 Nov 16 Posts: 11 Credit: 12,880,500 RAC: 0 |
This is a progress update for Hyper-V on Win 10 Pro and Ubuntu 20.04 as Virtual machine. I was to far away from linux world but I am retired so I have a lot of time to read documents and google :) Finally I managed to install ubuntu and configure it (with rsNeutrino's document links and google ). Also convert Virtualbox VMs win 10 and Win 7 managed to run under hyper-v but some lisence problems occur so left them on Virtualbox. As a result run the first WUs on Universe@home with Universe BHspin v2 v0.19 works. This is a huge gain for process time. Before runtime takes approximately 8000+ to 11000 seconds on win 10, now under hyper-v linux run times takes 2300-2500 seconds. This is really a mind-blowing operation. I am to happy to try this. Now I will try other projects to see what happen under linux. Thanks again rsNeutrino for giving these helpful ideas. Seti@home Classic account User ID 955 member since 8 Sep 1999 classic CPU time 539,770 hours |
Send message Joined: 13 Sep 15 Posts: 6 Credit: 21,290,767 RAC: 0 |
Absolutely correct. If you do that, Boinc will tell you when it starts that there is no virtualization Hence, it wont use Vbox, hence other projects using Vbox won't run. it's a choice to make. Good info overall rsNeutrino ! Thank you |
Send message Joined: 18 Jul 17 Posts: 138 Credit: 1,379,173,617 RAC: 0 |
I just got a Linux 20 system up with some spare threads. It is now crunching 5 CPU threads of U@H. The rest of the CPU threads are busy pushing gpus over at Einstein@Home. https://universeathome.pl/universe/show_host_detail.php?hostid=585565 Tom M A proud member of the OFA (Old Farts Assoc.) |
Send message Joined: 10 May 20 Posts: 310 Credit: 4,733,484,700 RAC: 0 |
Thanks Tom, every little bit helps with the team. A proud member of the OFA (Old Farts Association) |
Send message Joined: 11 Dec 20 Posts: 2 Credit: 15,953,633 RAC: 0 |
So this is pretty incredible. I spun up a VM using HyperV on my windows10 machine (which was already running crunching) and the performance seems incredible. 41 work units in about 6 hours? Seems amazing compared to windows 10. is the difference *really* that large? -jcp- |
Send message Joined: 10 May 20 Posts: 310 Credit: 4,733,484,700 RAC: 0 |
Yes, if you compare your validated results of your wingmen running native Windows, the speedup using a Linux kernel and recent glibc library of 2.31 or better really does run that much faster. A proud member of the OFA (Old Farts Association) |