Feature/tool request: split optimized plot files in to 4096 scoop files
Dear Burst developers,
Based on my understanding of plotting and mining it should technically be possible to split optimized plot files into 4096 individual scoop files and mine from them without any decrease in performance.
This would greatly improve portability and make it possible to move a 3TB plot to a 2TB and 1TB HDD for example. Also more file systems could be used (like FAT32 with a file size limit of 4GB). When moving plots between storage locations (for whatever reason) a failed transfer is pretty much disastrous and can cost you a lot of time (to transfer the whole plot file again) but if a single scoop file fails you can just continue where you left of without too much hassle. For me (and I hope for other users as well) this type of portability would be very useful. Lastly, it would be possible to temporarily make space on a HDD if needed (e.g. by moving a few scoop files to a USB drive) without having to delete the entire plot file. I think you could even improve performance if you have a bunch of smaller plot files by distributing scoop files evenly across hard drives for example.
When used properly just splitting the plot file into individual scoop files there should be no decrease in (mining) performance at all so I can't see any disadvantages but if there are any I would like to hear them (just make sure you don't put the same scoop number from different plot files on the same drive).
Details of proposal
Proposed file naming (add scoop number to the end of the plot filename):
I guess the splitting itself would be quite easy for anyone with basic programming knowledge (so not for me unfortunately) as optimized plot files can literally be split into 4096 equally sized files if I understand it right so I really hope Burst developers can see the added value of these split plot files. Of course there are tools (such as HJ-Split) that can already do this but as file naming is important with plots a dedicated tool would be better I think.
Features that would be nice to see in a "plot to scoop splitter" tool would be:
- Extract individual scoops based on number or range (e.g. just extract scoop 2834 or first extract scoops 1-1024 from a 4TB plot to fill a 1TB drive, then extract 1025-4096 to fill a 3TB drive).
- Quick split plot to same drive even when drive is full, so without rewriting all data. Maybe by rewriting the file index while leaving file data untouched so the single plot file shows up as 4096 individual scoop files (not sure if this is technically possible though).
Thank you for your time and I would love to hear your thoughts!
Bump! ( I want this feature. ) if it can actually happen
@jant90 It's a nice idea, but 4096 files of specific scoops is not going to be any faster than a regular optimized plotfile.
- Seek to start of scoop X in the Nonces, consecutively read the next X scoops.
Its the same in an optimized file or 4096 scoop files.
@haitch true, but it's more about the other advantages (mainly portability I guess) for me to be honest anyway.
Whene creating your plots just plot 1 nonce or how ever many GB of nonces you want. then you can move the files as you wish
@manfromafar But then mining is unoptimized and thus inefficient. When splitting an optimized plot file into 4096 scoops you still have the advantages of mining on an optimized plot.
@jant90 Your understand of space usage is slightly off.
Creating split nonce plots means you would have 4096 files each increment by 32 bytes for each nonce it holds, therefore your lower limit is 32 bytes while your upper limit is still limited only by your HDD space. I read your reasons for asking for it but as Mr. Wonderful would say, "I'm out"
@jant90 How does it become more portable? And if you're mining multiple drive equivalents into these 4096 files, you lose the parallelism efficiencies.
Well, portable as in:
- I want to move my 3TB plot from my PC to my NAS over LAN. After 8 hours of transferring the plot file its at 75% and the LAN connection is shortly interrupted. Transfer fails and I can start all over and 8 hours are lost (with split plots you could pick up where it left off).
- I change my mind and want to move my 3TB plot to a 2TB and 1TB disk (currently not possible).
- I temporarily need some extra space on my local HDD so I move over 5 scoop files to a temporary location and move them back later when I don't need the space myself anymore.
I guess for most people it's not really useful to have split plot files but people starting with mining might realize at a later moment they want to change their mining rig without replotting. With these large plot files that's just a bit hard.
I don't expect this to be realized but I just figured I would shoot the idea out there and see if people would be interested. Thanks for the replies though!
@jant90 why would anyone have 3.5 TB sized plot files in the first place ? Once you have >250 GB sized files, there is no increase in read speeds while mining. (for me, this may be different on other filesystems).
I have mostly 1 TiB files for ease of plotting (and 7 fit nicely on an 8 TB drive), and when needing space I temporarily delete one, and replot later when space is free again.
For splitting plotfiles into scoop-based files you have to have adapted miners. I don't see the point in having another set of parameters to be mastered. Most beginners are overwhelmed with the technical knowledge requirements, anyway. I mean, they read of "optimizing", "fast GPU plotting", immediately "want it!" and you see what is happening in the support section.
@vaxman So I shouldn't have done this?
06/23/2017 02:10 PM 6,000,268,541,952 13919803089879865906_0_22889208_22889208
06/23/2017 10:43 PM 8,000,678,920,192 13919803089879865906_50000000_30520168_30520168
06/24/2017 03:44 AM 8,000,678,920,192 13919803089879865906_100000000_30520168_30520168
06/24/2017 03:41 AM 8,000,678,920,192 13919803089879865906_150000000_30520168_30520168
06/24/2017 08:41 AM 8,000,678,920,192 13919803089879865906_200000000_30520168_30520168
06/25/2017 01:35 AM 8,000,678,920,192 13919803089879865906_250000000_30520168_30520168
06/26/2017 05:00 AM 8,000,678,920,192 13919803089879865906_300000000_30520168_30520168
06/26/2017 05:22 AM 8,000,678,920,192 13919803089879865906_350000000_30520168_30520168
06/27/2017 02:08 PM 8,000,678,920,192 13919803089879865906_400000000_30520168_30520168
06/27/2017 09:00 AM 8,000,678,920,192 13919803089879865906_450000000_30520168_30520168
06/28/2017 03:48 PM 8,000,678,920,192 13919803089879865906_500000000_30520168_30520168
06/30/2017 05:27 PM 8,000,678,920,192 13919803089879865906_550000000_30520168_30520168
06/29/2017 07:50 PM 6,000,268,541,952 13919803089879865906_600000000_22889208_22889208
07/02/2017 08:38 AM 6,000,268,541,952 13919803089879865906_650000000_22889208_22889208
06/28/2017 08:43 AM 5,000,080,130,048 13919803089879865906_700000000_19073792_19073792
07/08/2017 10:13 PM 4,999,937,523,712 13919803089879865906_750000000_19073248_19073248
07/02/2017 08:53 PM 5,000,080,130,048 13919803089879865906_800000000_19073792_19073792
07/03/2017 07:45 AM 5,000,080,130,048 13919803089879865906_850000000_19073792_19073792
07/05/2017 04:54 PM 5,000,080,130,048 13919803089879865906_900000000_19073792_19073792
06/22/2017 07:39 AM 5,000,080,130,048 13919803089879865906_1000000000_19073792_19073792
07/10/2017 09:22 AM 8,000,678,920,192 13919803089879865906_1050000000_30520168_30520168
06/27/2017 02:03 PM 8,000,678,920,192 13919803089879865906_1100000000_30520168_30520168
06/27/2017 02:03 PM 8,000,678,920,192 13919803089879865906_1150000000_30520168_30520168
So I shouldn't have done this?
Noooo ! Leave a tiny corner in the global space for me..THATS why I'm down to 1 block/day.
And an impressive creation date trail..and all CPU-plotted I guess..wow.
Hats off, gentlemen, this is plotting on industrial scale.
@vaxman Pennywise is a monster .....
This post is deleted!
@Han-Solo that seems pretty useless because you can just as well plot 1/40th and spent a lot less time on plotting.
This post is deleted!
@Han-Solo this only makes sense if the chances to win a block are not linear w/r to absolute plotsize;
Do 100 TB mine less than a tenth of 1 PB.
The actual code mods for plotting/mining are "trivial"..go ahead, the source is there.
This post is deleted!
@vaxman Why You have to answer this way? You feel smart now?
Idea was not to game the system or to get any advantage. I am not Pajeet. As I point already it make sense because the drives will work far more less = less energy spent = more profit.
I don't want more chances to win a block. Just to save energy. This is the whole point. Maybe for You it doesn't. But if You have hundred's of HDD's then is another story.
If You can't engage in a discussion at least don't be a dumb ass. Or whatever. Just go ahead as You are...
P.S. I am not a programmer. So how can be trivial?
Didn't mean to step on your foot.
But still, why not batch-control mining to the whee hours instead of messing with the code.