Feature/tool request: split optimized plot files in to 4096 scoop files

jant90

Dear Burst developers,

Proposal
Based on my understanding of plotting and mining it should technically be possible to split optimized plot files into 4096 individual scoop files and mine from them without any decrease in performance.

Benefits
This would greatly improve portability and make it possible to move a 3TB plot to a 2TB and 1TB HDD for example. Also more file systems could be used (like FAT32 with a file size limit of 4GB). When moving plots between storage locations (for whatever reason) a failed transfer is pretty much disastrous and can cost you a lot of time (to transfer the whole plot file again) but if a single scoop file fails you can just continue where you left of without too much hassle. For me (and I hope for other users as well) this type of portability would be very useful. Lastly, it would be possible to temporarily make space on a HDD if needed (e.g. by moving a few scoop files to a USB drive) without having to delete the entire plot file. I think you could even improve performance if you have a bunch of smaller plot files by distributing scoop files evenly across hard drives for example.

Disadvantages
When used properly just splitting the plot file into individual scoop files there should be no decrease in (mining) performance at all so I can't see any disadvantages but if there are any I would like to hear them (just make sure you don't put the same scoop number from different plot files on the same drive).

Details of proposal
Proposed file naming (add scoop number to the end of the plot filename):
12345678901234567890_0_11403264_11403264_0001
12345678901234567890_0_11403264_11403264_0002
...
12345678901234567890_0_11403264_11403264_4096

I guess the splitting itself would be quite easy for anyone with basic programming knowledge (so not for me unfortunately) as optimized plot files can literally be split into 4096 equally sized files if I understand it right so I really hope Burst developers can see the added value of these split plot files. Of course there are tools (such as HJ-Split) that can already do this but as file naming is important with plots a dedicated tool would be better I think.

Features that would be nice to see in a "plot to scoop splitter" tool would be:

Extract individual scoops based on number or range (e.g. just extract scoop 2834 or first extract scoops 1-1024 from a 4TB plot to fill a 1TB drive, then extract 1025-4096 to fill a 3TB drive).
Quick split plot to same drive even when drive is full, so without rewriting all data. Maybe by rewriting the file index while leaving file data untouched so the single plot file shows up as 4096 individual scoop files (not sure if this is technically possible though).

Thank you for your time and I would love to hear your thoughts!

HiDevin

Bump! ( I want this feature. ) if it can actually happen

haitch

@jant90 It's a nice idea, but 4096 files of specific scoops is not going to be any faster than a regular optimized plotfile.

Seek to start of scoop X in the Nonces, consecutively read the next X scoops.

Its the same in an optimized file or 4096 scoop files.

jant90

@haitch true, but it's more about the other advantages (mainly portability I guess) for me to be honest anyway.

manfromafar

Whene creating your plots just plot 1 nonce or how ever many GB of nonces you want. then you can move the files as you wish

jant90

@manfromafar But then mining is unoptimized and thus inefficient. When splitting an optimized plot file into 4096 scoops you still have the advantages of mining on an optimized plot.

IceBurst

@jant90 Your understand of space usage is slightly off.

Creating split nonce plots means you would have 4096 files each increment by 32 bytes for each nonce it holds, therefore your lower limit is 32 bytes while your upper limit is still limited only by your HDD space. I read your reasons for asking for it but as Mr. Wonderful would say, "I'm out"

-IceBurst

haitch

@jant90 How does it become more portable? And if you're mining multiple drive equivalents into these 4096 files, you lose the parallelism efficiencies.