well i'm still evaluating the best solutions, but i think i will start next week. I still need some more infos about opencl, but i know someone who is very experienced in gpu computing...
Best posts made by tco42
-
RE: GPU Plotting (Direct mode) - Why does it suck so bad?posted in Mining & Plotting
-
RE: GPU Plotting (Direct mode) - Why does it suck so bad?posted in Mining & Plotting
If you look at the code you can see that there are some parts, that are not optimal like generating the plots and writing to hdd is not done in parallel. In direct mode writing takes even longer because you dont write once per staggersize but 4096 times (including 4096 seeks).
It lowers my nonces / min from around 200k when plotting a single 128gb file in ram (with 128gb stagger size) to less than 50k while plotting several 8tb hdds in parallel.
There will soon (4-6 weeks?) be a newer version of the gpu plotter that is more efficient but requires a bit more ram
-
RE: GPU plot generator v4.1.1 (Win/Linux)posted in Plotter
yeah, thats a part of the problem... lets say your gpu generates 24,000 nonce/min (equals 100mb/s) and your hdd can write 100mb/s so 1tb should take about 10,000 seconds in reality it looks like that (of example at 4gb stagger size):
- gpu generates 4gb of data in 40 seconds
- hdd writes 4gb of data in 40 seconds
- gpu generates 2. set of 4gb of data in 40 seconds
- hdd writes 2. set of 4gb of data in 40 seconds
etc.
-> it will take 20,080 seconds to finish
if you add additional ram to hold a second set of data it looks like that:
- gpu generates 4gb of data in 40 seconds
- hdd writes 4gb of data while gpu generates the 2. set of data in 40 seconds
- hdd writes 4gb of data while gpu generates the 3. set of data in 40 seconds
etc
-> it will only take 10,080 seconds to finish (about 100% faster)
if your gpu is faster than your hdd write speeds you would still be able to keep your hdd writing 100% of the time so 48,000 nonce/min and 100mb/s write speed would look like that:
- gpu generates 4gb of data in 20 seconds
- hdd writes 4gb of data (40s) while gpu generates the 2. set of data (20s) and waits (20s) in 40 seconds
- hdd writes 4gb of data (40s) while gpu generates the 3. set of data (20s) and waits (20s) in 40 seconds
etc.
-> it will take 10,060 seconds to finish
EDIT:
a faster gpu would ofcourse change the first example to 40s hdd and 20s gpu which results in 15,060 seconds so the optimized code would be about 33% fasterEDIT 2:
you should be able to optimize the plot while its in system ram and waiting to be writen without any additional time -
RE: GPU plot generator v4.1.1 (Win/Linux)posted in Plotter
Hi,
is this project still being developed?
I'm not really good at C++ since most of my work is based on Java but i think there are still some points that could be improved. I looked at my gpu and hdd usages and found it very odd to have like 50% average gpu usage and 50% average hdd usage becase generation and writing are not done in parallel...For example it should be possible to add two features:
- hybrid mode, which does fill a part of the hdd (like direct mode does for the full plot size) while the gpu calculates the plot and then write it
- reserve twice (4x?) as much system ram then generate part after part (keeps the gpu running all the time) while the hdd writes those parts
-
RE: GPU plot generator v4.1.1 (Win/Linux)posted in Plotter
I forgot to mention that i'm mining the first 24*8tb (NTFS formatted, some 8tb plot files, some 1tb plot files) at the same time. Reading speeds are around 4gb/s, roundtimes below 15s even while plotting
-
RE: GPU plot generator v4.1.1 (Win/Linux)posted in Plotter
@vaxman
i spent the few days trying your suggestions, but it didn't matter how many plots were written to the ssd, it was the same speed (about 200k/min) all the time. Its the same like plotting to ram, since its a quite powerful samsung 960 pro with writing speeds exceeding 2gb/s.When the next 24 hdds arrived i tried switching to linux (ubuntu 17.04) and ext4 filesystem and it seems like i was right to blame the problems on windows.
- ext4 does not need to create the plot file before starting to write to it. this saves about 12-15h at the start of the plot
- linux seems to handle those threads much better than windows. with the same settings and the same hardware my nonces/min stay at around 210-220k.
- it does not matter how many plot files are used (as long as there are more than 5, which are required to get enough write speed) the nonce/min stay at above 210k
After testing for some hours i completely wiped the new hdds and started plotting around 10h ago. Currently 20% are done and there are less than 40h remaining while plotting 24*8tb.
-> Switching from windows (120h for 12 * 8tb) to linux (50h for 24 * 8tb) is a very good choice if you want to plot in direct mode or to many drives at the same time :)
-
RE: GPU Plotting (Direct mode) - Why does it suck so bad?posted in Mining & Plotting
After several days of analysing the code and even more time spent testing changes i found the solution to my problems.
There seems to be a problem in windows and its thread management which causes some of the threads not to run at the perfect time. Simply switching to linux (ubuntu 17.04) allowed me to keep plotting at over 200.000 nonce/min with exactly the same settings and exactly the same code.
Also if you choose ext4 as filesystem you dont need to create the empty plot file at the beginning.Looks like there is no need to change anything in the code.
-
RE: GPU plot generator v4.1.1 (Win/Linux)posted in Plotter
@BeholdMiNuggets said in GPU plot generator v4.0.3 (Win/Linux):
@tco42 said in GPU plot generator v4.0.3 (Win/Linux):
- those discs (hgst deskstar 8tb) are all connected to a 24 port sas hba.
Do you find the speed difference between SAS/SATA makes any difference for Burst Mining? And what about the Hard-Drive Cache?
(https://www.broadcom.com/products/storage/host-bus-adapters/sas-9305-24i).
Only see 12x data slots on that [PCIe 8x] DIC. Do they all double up? Presume you also have a dedicated (rack) server case to store all those (24) drives. Any pictures?
SAS or SATA makes no difference in this case, because i use SATA drives and SAS->SATA adapter cables.
There are 6 SAS connectors on the broadcom and each connector has 4* SAS3 connections. My "professional" rack case is selfmade :D will post pictures soon
i tested the maximum speeds of the hba by copying from my first 12 hdds to the second set of 12 hdds which resulted in about 200mb/s per drive -> 2,4gb/s total read and write with all hdds connected to the same hba.
I guess that's why you need Server CPUs. What's the MoBo & Psu?
You can use the same PCIe adapter in every mobo with a PCIe x8 slot. It does not need too much gpu power. Main reason to go for xeons is, that i get 2*40 PCIe lanes which are quite useful when i use that server as a workstation/gaming pc when mining is no longer profitable.
- I'm currently using ntfs
Is there a possible, superior option that you're considering?
EXT4 could be useful when switching from windows to linux.
those plots were just some tests to find the bottleneck that limits my plotting.> you dont need to connect the gpus using sli, there are just minor differences. those gpus should be able to put lots of data to the drives (200k nonce / min = 50gb / min = 20 min/tb). But there is a bottleneck (mb in the plotter, mb in windows) that i'm still trying to find.
Fortunately, each HDD only has to be plotted once or twice. So, not that burdensome considering the projected working life of the Drives.
Yeah, thats a big plus. But i still needed about 120h to plot only 12 drives (96tb) and this server might get upgraded to more than a pb soon :)
-
RE: GPU plot generator v4.1.1 (Win/Linux)posted in Plotter
@vaxman im still not sure why there is such a low gpu usage at my setup...
im currently using 2* 1080ti, 2* xeon E5-2620, 128gb ram and 12* 8tb hddsi can get more than 200,000 nonce / min when plotting to a single samsung 960 pro which should be the maximum those gpus can calculate. when plotting to those 12 hdds my speed drops to about 50.000 nonce / min. It seems like the gpus are only working when the hdds finished writing and the hdds are only writing when the gpus finished calculating.