Filled up half a PetaByte. Now What?



  • @Garbear What cpu and ram do you have?



  • @ChuckNorris Yeah optimizing will see a big gain in read speeds. It's been a while since I've plotted unoptimized then optimize them as I use xplotter now but if I remember it was close to double scan speeds after the optimizing.



  • @ChuckNorris My current mining rig is a asrock extreme 3 gen 3 with i5 2500k stocks speeds with 16gigs ddr3 ram and a GTX 960.

    My new plotting and mining rig i'm still getting setup is a ASRock Fatal1ty X370 Professional Gaming AM4 AMD X370, ryzen 7 1700 and 16 gigs ddr4 ram. Plotting with xplotter atm with 12 of 16 threads is a little over 15k nonces/min

    It's got 10 sata3 ports, 8 usb3 ports and 3 pci express x16 slots for my usb3 controller cards.



  • @ChuckNorris said in Filled up half a PetaByte. Now What?:

    But you do understand how costly and how much more of a mess it is to have 4 drives per hub when you have this many hard drives.

    I bought Seagate Backup Plus drives, since they had an integrated 2-port hub on them. I daisy chained multiple groups of HDDs and never had to buy a single hub in addition. :)



  • Half petabytes 500tb. am i right? Massive mining.



  • @ChuckNorris Half a PB? Outstanding! Very, very impressive. Now go find a wall and hit your head against it, really hard:))

    You should have plotted AND optimized. It will take you an insane amount of time to optimize those plots in-place. More RAM will help, sure, but the bottleneck with your setup will be I/O. Lessons learned, you'll do better on the next go:)



  • @ChuckNorris said in Filled up half a PetaByte. Now What?:

    Would these things help my count speeds?

    1. Switching from Kaby i5 to Kaby i7? And if so. K or not to K.
    2. upgrading from 16gb to 32gb of ram?
    3. How can i better utilize my gtx 1070?

    You seem to be "going mediaeval" on this burst thing! I'm fairly new to these parts, but have some suggestions. First, forget Kaby-Lake, Intel have jumped the shark. The Ryzen CPUs have done well, and now AMD are about to release their "ThreadRipper" range. In my (limited) experience, more threads does make a difference, and they will be offering a 16/32 variant at about half the price of the Intel /Xeon equivalents. You'll also need one of the new X399 Motherboards, every one of which sports 64x PCIe lanes. That means you can run a whole lot of SAS/SATA DiCs, extra GPUs etc. As for the other components, if you intend to expand further, then Ram it up as far as you can afford. And consider a 1600W Platinum PSU (or 2). Plus if you really have the €£$, then evaluate some high-capacity SSDs, just for plotting.



  • @illuminatus excuse me. Do you have a half peta setup. And no. Your wrong. If I plotted optimized plots off the bat. I only get 5000 nounces because they are SMR drives. Instead. I can buffer two drives at once and get 28000 nounces per drive. So for me to be able to fill up half a Peta in under 2 weeks. So I can start mining right away. I still get read times of 90 seconds with un optimized plots. Which is still pretty decent. Don't make me get my homies to back me up. Maybe do your research before insulting people.



  • @BeholdMiNuggets amds are garbage. I plot and count with gpus



  • @ChuckNorris said in Filled up half a PetaByte. Now What?:

    @BeholdMiNuggets amds are garbage. I plot and count with gpus

    You will swallow that words pretty bad when you will optimize your drives now! LOL
    You can only optimize with CPU from the best of my knowledge...
    So now for optimizing I would suggest a good CPU and RAM (the more RAM the better, to optimize)...
    If you will plot a lot more I would suggest you to do buffer plots with your GPU but each plot with the same number of nonces as what you used as stagger, because then you will not need to optimize them... ;D
    Then you will end with thousands of plot files but I heared @dawallet say (sometime ago) that you can concatenate plots, meaning that after you plot the thousands of plots you can join the plot files into one file (or how much you want), although I am pretty sure you will not need to do this, and if you need to do this I recommend you to ask @dawallet how can you concatenate plots in windows, because I never tried, but I've heared it's possible... ;D

    Maybe even worth, you replot disk by disk with the way I said above (GPU), with your GPUs maybe its better than using the CPU to optimize everything... Although you can also keep using the CPU to optimize it at the same time as you replot optimized plots with your GPUs... The faster the better ;P

    Good luck... 90 seconds for 500Tb of unoptimized plots I would say it's a nice time but not nice enough to keep it unplotted defenetly xD


  • admin

    @gpedro that was not me but I know you can only merge plot files with the exact same stagger.



  • @daWallet @gpedro I would like to merge my plots. They all have the same stagger size. How do I do that?


  • admin



  • @daWallet Hmm I wasn't sure it was you that said but I remember you was in the discussion about this and I knew you had informations... hehhehe ;D
    Thanks for sharing again, I am sure the other discussion is somewhere buryed here in the forum and I couldn't find it xD



  • @daWallet thanks. I am afraid though that in that way there is a need to have extra space for the new merged file, so I don't see how I can do that. All my plots are generated with Xplotter, so also not sure about the sequential issue. Perhaps @Blago could give his input here :)



  • Just did a test with two small plot files and it seems to work fine. but as I said the problem here is the space. Don't know if there is another way to generate the merge file without having to preserve a copy of the original ones...



  • @vExact So it created a duplicate file including the two files and kept the old ones?
    Now your new file is not optimized right?
    Did you had to run PlotChecker after for mine with it?
    Your 2 files had sequential nonces?
    Sorry for the questions but I guess its what you get by being pioneer hahahah ;P



  • @gpedro I guess as the original files are optimized from beginning the merged one should be optimized too, although I might be wrong on that one. I just assigned the corresponding name it would have if generated as a single plot file. I guess I should run plot checker to verify it is correct in any case.

    0_1497296722858_b63ac648-0920-414b-8576-9d028ce12173-image.png



  • @gpedro PlotChecker tells me the file is ok. So it seems to work with no problem :)
    Just have to type e.g.

    >copy /b *. 10783921033877668933_0_8192_8192.
    

    But still the problem of the space remains, so I think this is unfeasible.



  • @ChuckNorris Consider this - my buddy and I both have ~100TB rigs, with roughly equivalent PC specs. All of my plots are optimized and all of his are not, but both of our rigs get scan times around 32 seconds with jminer and a decent GPU. Both PCs use about 50% of the i5 or FX-8350 CPU, but the difference is that his PC ends up using ALL of the 16GB of system RAM, whereas mine uses about 8GB total. It seems that unoptimized plots are not that big of a bottleneck if you have enough hardware to deal with it, which makes me think most of your slow scan times are due to having too many drives per USB3.0 controller and/or not having adequate hardware. Upgrading to an i7 and doubling the memory might help, but bang for the buck starts to play a major factor at that point. In your situation, I would build a third machine and spread the hard drives out over all three. A fourth machine could be dedicated to plotting and then be repurposed to mine Ethereum or Zcash when it's done.

    I'm curious - how much RAM is being used in your machines each round? I'd suspect just about all of it.

    For shits and giggles, pull out 4 drives from each 7 port hub and see what happens to your scan times.


Log in to reply
 

Looks like your connection to Burst - Efficient HDD Mining was lost, please wait while we try to reconnect.