80TB Plotting and Optimise more efficient way?



  • Hi, guys

    What you would recommend to plotting and optimise 80TB?
    They are separated into, x2 WD of 16TB (two internal HD of 8T ) and x6 Seagate of 8TB. There are 10HDs of 8TB. I have seen several videos but they are to basic.

    Is it a good idea to plot all of them at the same time on the same computer? OR can I do this on different computers? And then at the end just plug back all into the main computer? The Burst account recognise the plotts I need re-link then or something similar?

    I have a rig and I can use it for faster GPU plotting/optimisation. For example plotting each HD on a separate GPU, that is possible?
     
    There are 10HD of 8TB, optimising them in the following order: 5HD optimised for the other free 5HD , then 2HD in to the free 2HD and I will get a one HD without optimisation.

    I did not found any information about managing large amounts of hard drives in a more efficient way, if someone can help, Thanks.


  • admin

    @Rodrigogmail said in 80TB Plotting and Optimise more efficient way?:

    Hi, guys

    What you would recommend to plotting and optimise 80TB?
    They are separated into, x2 WD of 16TB (two internal HD of 8T ) and x6 Seagate of 8TB. There are 10HDs of 8TB. I have seen several videos but they are to basic.

    WD 16TB has 2x8TB WD Red i guess? You should use them as single drives, not as RAID!
    You should plot to this drives ONLY and not to the Seagate, as the WD drives are PMR and Seagate are SMR (If they are Archive drives)
    https://forums.burst-team.us/topic/2394/shingled-magnetic-recording-smr

    Is it a good idea to plot all of them at the same time on the same computer? OR can I do this on different computers? And then at the end just plug back all into the main computer? The Burst account recognise the plotts I need re-link then or something similar?

    You can plot on multiple machines, just ensure your plotfiles do not overlap.
    https://forums.burst-team.us/topic/288/plots-101/5

    I have a rig and I can use it for faster GPU plotting/optimisation. For example plotting each HD on a separate GPU, that is possible?

    You can use one GPUPlotter instance to plot multiple drives at once ... you can provide multiple GPUs to the single GPUPlotter instance via devices.txt
    You can use latest GPUPlotter https://forums.burst-team.us/topic/58/gpu-plot-generator-v4-1-1-win-linux in direct mode or Xplotter (CPU) ... to create optimized plotfiles directly.

    There are 10HD of 8TB, optimising them in the following order: 5HD optimised for the other free 5HD , then 2HD in to the free 2HD and I will get a one HD without optimisation.

    I suggest create optimized plotfiles directly. Like mentioned above.

    I did not found any information about managing large amounts of hard drives in a more efficient way, if someone can help, Thanks.

    Conclusion:
    Plot the WD red drives with GPUPlotter in direct mode ... maybe multiple drives at once, depending on your GPU(S) ... if finished copy over that plotfiles to Seagate Archive drives and repeat.
    (Ensure the plots created on WD are not too big for Archive drives, few bytes could make the idea not working)
    At same time or alternative you could use XPlotter with CPU.



  • @luxe Thanks I'm updating the processes



  • @luxe It's clear that I'm no expert on the subject, I tried to read some documentation that is very few without success. Inexperienced users with truncated information between expert opinions, it became complicated something super simple.

    Anyway
    I think the safest process would be to connect all the HDs on an I7 computer and plot directly to each HD without making copies or move files. Using XPlotter. Will XPlotter automatically "see" the plots and not overlap them?

    Or I have to manually put the plots values. (eg. divide by 4 the recommended values) I'm confused in this part, because all the explanations I found only shows how to make one plot and not several

    Do you think I will have problems if I start the process in one HD and soon after starting the processes in others so on? You are helping a lot.



  • @Rodrigogmail You can run multiple instances of xplotter concurrently with no problem. What i7 do you have (how many threads does it have)? For each instance, you'll just need to manage the # of threads and amount of RAM used. E.g. if you have 8 threads and 16gb RAM, I'd suggest plotting 3 or 4 drives in parallel, each with 2 threads and 2-4gb of RAM.

    For nonces, you need to tell xplotter what nonce to start at, but not the number of nonces to plot:

    XPlotter_avx.exe -id 16098533444009108472 -sn 287305992 -n 0 -t 4 -path N:\plots -mem 12G
    

    For each new drive you start, you'll need to adjust the -sn parameter so it starts on the nonce after your last plotted one. If you want to simply fill the entire drive as much as possible, leave the -n parameter as 0.

    Something like the below will be your sequence:
    Plot1 config = XPlotter_avx.exe -id 16098533444009108472 -sn 0 -n 0 -t 4 -path D:\plots -mem 12G
    Plot1 name = 16098533444009108472_0_7629376_7629376

    plot2 config = XPlotter_avx.exe -id 16098533444009108472 -sn 7629376 -n 0 -t 4 -path E:\plots -mem 12G
    Plot 2 name = 16098533444009108472_7629376_15260544_15260544

    Thus, the -sn for plot 3's config will be (plot2's startNonce + plot2's #ofNonces) = 7629376 + 15260544



  • @IncludeBeer This business is getting serious lol.
    You're right I7 of 8 threads 16G of RAM.

    It would be an absolute success on Youtube, a UPDATED video of how to plot 1 and several HDs runing multiple instances with XPlotter, there is nothing like it.

    I'll put another explanation here to help ... I'm sure I'll do some shit

    Buzz1
    So do you do the same process for each drive? Say i have 5 8TB Drives, don't i have to do something else so the drives don't overlap?
    Medicine Hat Computers
    Medicine Hat Computers1 week ago
    Yes your right you must adjust the starting block to one after the last block on the last hard drive. NOTE: Plotter may adjust the blocks so in the cmd take note of the adjustments the program made and edit your other bat file according to the block number . Hope this helps if you need more help feel free to ask.
    Buzz1
    So i have 5 8TB drives, i used the nonces calculator. My drives actually show 7.27 TB. So if i put 7277 i get 29806592. So can you give me an example how it should look with the 5 drives? I am trying to understand the setup with multiple drives. Thanks!
    Medicine Hat Computers
    Hey so your first drive nonce will start at 0 and end with 29806592 now take note of your end number start your plotter for your first drive. Once it starts the program may adjust the nonces for the ending number take note of it. Now for the second drive calculate the new numbers you will start at lets say the program didn't adjust it but im sure it will you will start at 29806593 and end with 29806592 note the program will adjust the ending number so the plots will not overlap thats why you must take note of the adjustments the program makes and add one integer to the end number for the second drive. or have fun waiting for hours to find out its wrong only to redo the plots/. Hope this helps
    Read more
    Buzz1
    Ok i got it going and my end is 32700000. So the next start on next drive will be 32700001? and so on? I thought i saw someone doubling size on next hdd maybe i was misaking. Also, i have multiple rigs pointing to same bust account. Do i need to consider those in the chain or are they separate? Like is it possible i am overlapping between PCs? Thanks for this info. Very helpful!
    Medicine Hat Computers
    ya you bet . it should only write the size you inputted into the bat file from the calculator. Let me know how it goes cuz im limited here with equipment and its nice to know it will work with big drives as well. Oh maybe ive thought of that exact scenario I assume yes if your other rigs are plotted from the same nonces as these drives you may need to redo them but that just a theory that i think would happen .
    Reply


  • admin

    @Rodrigogmail I plot multiple drives simultaneously, jumping 50M nonces per drive. It leaves gaps, but it the number of nonces you have, gaps don't really matter.

    @setlocal
    @cd /d %~dp0
    start /low XPlotter_avx.exe -id 13919803089879865906 -sn 0 -t 8 -path c:\b\00 -mem 8G
    start /low XPlotter_avx.exe -id 13919803089879865906 -sn 50000000 -t 8 -path c:\b\01 -mem 8G
    start /low XPlotter_avx.exe -id 13919803089879865906 -sn 100000000 -t 8 -path c:\b\02 -mem 8G
    start /low XPlotter_avx.exe -id 13919803089879865906 -sn 150000000 -t 8 -path c:\b\03 -mem 8G
    start /low XPlotter_avx.exe -id 13919803089879865906 -sn 200000000 -t 8 -path c:\b\04 -mem 8G
    start /wait /low XPlotter_avx.exe -id 13919803089879865906 -sn 250000000 -t 8 -path c:\b\05 -mem 8G
    start /low XPlotter_avx.exe -id 13919803089879865906 -sn 300000000 -t 8 -path c:\b\06 -mem 8G
    start /low XPlotter_avx.exe -id 13919803089879865906 -sn 350000000 -t 8 -path c:\b\07 -mem 8G
    start /low XPlotter_avx.exe -id 13919803089879865906 -sn 400000000 -t 8 -path c:\b\08 -mem 8G
    start /low XPlotter_avx.exe -id 13919803089879865906 -sn 450000000 -t 8 -path c:\b\09 -mem 8G
    start /low XPlotter_avx.exe -id 13919803089879865906 -sn 500000000 -t 8 -path c:\b\10 -mem 8G
    start /wait /low XPlotter_avx.exe -id 13919803089879865906 -sn 550000000 -t 8 -path c:\b\11 -mem 8G
    start /low XPlotter_avx.exe -id 13919803089879865906 -sn 600000000 -t 8 -path c:\b\12 -mem 8G
    start /low XPlotter_avx.exe -id 13919803089879865906 -sn 650000000 -t 8 -path c:\b\13 -mem 8G
    start /low XPlotter_avx.exe -id 13919803089879865906 -sn 700000000 -t 8 -path c:\b\14 -mem 8G
    start /low XPlotter_avx.exe -id 13919803089879865906 -sn 750000000 -t 8 -path c:\b\15 -mem 8G
    start /low XPlotter_avx.exe -id 13919803089879865906 -sn 800000000 -t 8 -path c:\b\16 -mem 8G
    start /wait /low XPlotter_avx.exe -id 13919803089879865906 -sn 850000000 -t 8 -path c:\b\17 -mem 8G
    start /low XPlotter_avx.exe -id 13919803089879865906 -sn 900000000 -t 8 -path c:\b\18 -mem 8G
    start /low XPlotter_avx.exe -id 13919803089879865906 -sn 950000000 -t 8 -path c:\b\19 -mem 8G
    start /low XPlotter_avx.exe -id 13919803089879865906 -sn 1000000000 -t 8 -path c:\b\20 -mem 8G
    start /low XPlotter_avx.exe -id 13919803089879865906 -sn 1050000000 -t 8 -path c:\b\21 -mem 8G
    start /low XPlotter_avx.exe -id 13919803089879865906 -sn 1100000000 -t 8 -path c:\b\22 -mem 8G
    start /wait /low XPlotter_avx.exe -id 13919803089879865906 -sn 1150000000 -t 8 -path c:\b\23 -mem 8G

    this starts 6 drives plotting at once, the /wait on the sixth drive stops the plotter until it completes its plot. So six drives start plotting, when drive #6 is complete, the next 6 start.



  • @haitch Thanks a lot, I am organising all this information (with the nick of the authors), and making a documentation in order to publish. The Hds arrived next week, I'll give some feedback for you guys.



  • @haitch
    Hds installed and working ...
    The WD 16TB does not allow me to work with the HDDs separately, but they are not in RAID.

    Would be like this:
    @setlocal
    @cd /d %~dp0
    start /low XPlotter_avx.exe -id "your ID" -sn 0 -t 8 -path h:\Burst\plots -mem 15G (8TB Hdd)
    start /low XPlotter_avx.exe -id "your ID" -sn 50000000 -t 8 -path i:\Burst\plots -mem 15G (8TB Hdd)
    start /low XPlotter_avx.exe -id "your ID" -sn 100000000 -t 8 -path j:\Burst\plots -mem 15G (8TB Hdd)
    start /low XPlotter_avx.exe -id "your ID" -sn 150000000 -t 8 -path k:\Burst\plots -mem 15G (8TB Hdd)
    start /low XPlotter_avx.exe -id "your ID" -sn 200000000 -t 8 -path l:\Burst\plots -mem 15G (8TB Hdd)
    start /wait /low XPlotter_avx.exe -id "your ID" -sn 250000000 -t 8 -path m:\Burst\plots -mem 15G (8TB Hdd)
    start /low XPlotter_avx.exe -id "your ID" -sn 300000000 -t 8 -path f:\Burst\plots -mem 15G (16TB Hdd)
    start /low XPlotter_avx.exe -id "your ID" -sn 350000000 -t 8 -path g:\Burst\plots -mem 15G (16TB Hdd)

    Volume Info
    alt text

    Thanks a lot you guys.
    I will make a video on YouTube - Plot for Dummies - simultaneous HDDs with XPlotter.


  • admin

    @Rodrigogmail You can use the drive letters, if you go over 24 drives though, you'll need to change to using mount points.



  • It's working!
    I'll try to make it even simpler. Because my English does not allow me to go deeper.
    Disregard the ""

    Open the CMD (DOS window) and go to the folder where XPlotter is and type:

    XPlotter_avx.exe -id "your ID" -sn "nonce number" -t "threads " -path "x": \ Burst \ plots\ -mem "RAM amount" G

    e.g.:
    XPlotter_avx.exe -id 13919803089879865906 -sn 0 -t 8 -path d:\ Burst \ plots\ -mem 8G
    XPlotter_avx.exe -id 13919803089879865906 -sn 50000000 -t 8 -path e:\ Burst \ plots\ -mem 8G
    XPlotter_avx.exe -id 13919803089879865906 -sn 100000000 -t 8 -path f:\ Burst \ plots\ -mem 8G
    XPlotter_avx.exe -id 13919803089879865906 -sn 150000000 -t 8 -path g:\ Burst \ plots\ -mem 8G
    XPlotter_avx.exe -id 13919803089879865906 -sn 200000000 -t 8 -path h:\ Burst \ plots\ -mem 8G

    • -id If you got here it's because you already know your ID from your wallet!
    • -sn "number" reference to start your nonce. It does not have to be a specific number but one that tells the XPlotter where to start, JUST THAT! As Haitch explained in the previous comment, "jumping 50M nonces per drive. It leaves gaps, but it's the number of nonces you have, gaps do not really matter" this fits any size of Hdd. This part is where I found it more difficult because many people still think one nonce should start exactly where the other stopped. NO!
    • -t search on Google how many threads your processor has put the number there.

    • "x" the address where the Plote will be created, need to be X: \ Burst \ plots, X is the letter of your Hdd
      With me it just worked with these rules.

    END!

    As a reference,
    I have an processor i7 with 8 threads and 16 of RAM, I use the totality of threads and the maximum memory possible to not let the system freeze, in order to plot a 8T hd. In 24 hours completed 13%. On another computer I have an Xeon server with 8 threads and 16 RAM and completed 8% in 24 hours, why the fuck? Same hd. I think it's because of USB 2.0

    At the end you can plug the Hdd into any computer to mine, but not in different accounts it has to be referring to your ID that you created the Plots. In addition to all hds apparently they should be connected on the same computer to a single account. Connecting to the same account on different computers is possible but I've seen problems. I need to research that part.
    You can plot on multiple computers at the same time, just follow the nonces rules.

    I'm not an expert I started this Burst business from scratch only 4 days ago. And I found it very difficult to find simple explanations for what is simple. People are really confused. Maybe because they are "kids" lol

    Remembering this is just to plot your HD in all your computer capacity and all the space available on your HD in order to finish it ASAP! If you have more than 24 hd the process is different.

    Please if I explained something wrong, correct me

    Thank you very much to luxe, IncludeBeer and haitch


  • admin

    @Rodrigogmail Well explained. To address some of your comments:

    • USB2.0 will really slow you down, try to go SATA/SAS/USB3

    • You can have drives plotted to different accounts, but there a couple of gotcha's

    • Blago will mine multiple different accounts simultaneously, but they all need to be assigned to the same pool. If different pools, you need to run an instance of blago per pool. jMiner requires you run a separate instance per address

    • Using the same account on different PC's is not an issue, just make sure all starting nonces are unique



  • some numbers UPDATED 29/06/2017

    • External Hdd 3.0 /7200 RPM/8Tb

    CPU Plot XPlotter 1.0

    • i7 with 8 threads and 16 of RAM - ~10K nonces/min (tested for 6 hours)
      Hdd Write average of 10MB/s
      one week to finish

    • Xeon server with 8 threads and 16 RAM - ~10K nonces/min (usb 2.0) (tested for 6 hours)

    GPU Plot gpuPlotGenerator 4.0.3 (4.1.1 do not work, [ERROR] Unable to extend output file: 112)

    • Nvidia 1070 X /8G - ~20K nonces/min (~96 hours to finish)
      Hdd Write average of 80MB/s with peaks of 100 and low of 60
      4 days to finish full tested
      FAIL!!!! Corrupt Plot

    Conclusion, although the GPU plot is 2x faster did not guarantee plot integrity.

    I do not know if I can get better numbers, only 1 week of tests. However based on what I read these numbers are expected to keep the system stable.

    Important
    Certainly depending on the size of your plot the plotting capacity will decrease over time. With that started at 20k once/min will probably reach half of this speed 50% when reach 50% of Hdd. On an 8TB Hdd

    On GPU plot devices.txt
    0 0 1024 128 8192
    I will translate in a simpler way, for us more limited people lol
    plataforme id 0/ divice id 0/ 1024 amount of GPU memory / 128 Must be less or equal GPU memory/8192 less if your sys craches or freezes.

    About 128 and not 8192
    For some reason I still do not know the GPU memory, in my understanding should be something close to the limit of the graphics card, in fact is far from it. If I increase it the system freezes.



  • Thanks for taking the time to explain your findings so far. As another beginner I found it really helpful.

    Happy Mining.



  • @Squares Know which is the biggest problem who develops the tools does not do the proper documentation and testing, I think to keep costs low. But this creates a huge confusion, 95% of the users are lay,amateurs,enthusiasts,kids. Learning through trial and error demands enormous effort and time. When I started few days ago I did not know that were 2 totally different processes GPU/CPU



  • @Rodrigogmail
    I'm a little lost on how to make the connection of all HD



  • @Rodrigogmail said in 80TB Plotting and Optimise more efficient way?:

    @Squares Know which is the biggest problem who develops the tools does not do the proper documentation and testing, I think to keep costs low. But this creates a huge confusion,

    You know, there are so many different configurations out there.
    And if you want to plot as fast as you can with your setup, you will have to put some effort into finding out the best parameter set for your setup. Quite simple.

    The technical background, and I do not mean Burst here, is complicated when you are not a technical person. We're talking about matching

    • your GPU's spec (global memory, private memory, number of compute units, number of actually usable compute units, when using this GPU for OS display radically reduce work load, ..)
    • your disk configuration (PMR/SMR have very different needs here, how fast can the disk(s) ingest data, what type of filesystem, caching issues)
    • your willingness to dedicate a machine to plotting vs. "I still need to work on that box!", RAM available, cores/threads/cycles available, ..

    This is way beyond the average users background.
    There is the All-In-One Wallet, and it works very well (on Windows). If you want to speed things up beyond that, you need to invest time and effort and learn something along the way.

    Take this with a grain of salt from someone that started plotting his first 10 Terabytes with the original, java-based plotter. This was 10x slower and needed humongous amounts of RAM - we've come a long way since then.

    tldr; stop whining, start to learn about your system.


  • admin

    @Rodrigogmail If you think the process today is difficult, you should have been here 3 years ago when I started. The state of Burst mining is so much easier today, Can it be better, yes - but compared to what we started on, today it's relatively easy.



  • In fact I understand the position of you guys, I'm an IT Manager for 25 years and I'm not a developer, but I have the notion of complexity to manage people to deal with. Reminding developers that user does not calculate or should know how much memory he has in kilobyte has on the computer is something repetitive.
    Who does it is the software, the information is already there. Make an application that the user needs to deal with dozens of parameters and variables, where the software itself can recognise and manage it, something is wrong with the project, come back and do it right. User only consumes.

    GPU plot almost hit, unfortunately the parameters do not work.
    Very good start my dog can do ... 0 0 3 9 Q ready!
    Then the disaster of having to play with guessing numbers and see what works ... implement a micro benchmark and you're done!

    We live in capitalist countries, it is normal to charge for your work, no problem in that. Good example is the Awesome Miner, it is far from being a great app, but with the money he earns he is improving very fast.

    Check this out, basic its what happening, quite normal http://www.techrepublic.com/blog/10-things/10-signs-that-you-arent-cut-out-to-be-a-developer/

    Lets see if this plot work 8Tb by 20K nonces/min in 30 hours...

    TIP
    On Device Manger/disk drives/ disk properties/policies/removal policy
    change to Better performance


  • admin

    @Rodrigogmail The tools are improving: xplotter will automatically calculate how much drive to plot, will use sensible thread/mem settings that you can override. It's not long since being able to plot/mine required intimate knowledge of the coin; tools like xplotter and the AIO have made it much better, and improvements will continue


Log in to reply
 

Looks like your connection to Burst - Efficient HDD Mining was lost, please wait while we try to reconnect.