Plotting and Mining on Amazon Cloud Drive
-
@machasm
I read somewhere that in order to read the plots the amount of data required to download is 0.00025 of the total value?
If so that would mean on a 30TB plot you would need to download 7GB?
Have I got that right??
-
@machasm Yes I have read .024% of the data being read. Not certain of the maths, may have got the noughts wrong, but doesn't that work out to 720MB for 30TB Plot?
Rich
-
@machasm its 1/4096 th of the file size = 0.000244140625
30TB = 30000GB
30000GB * 0.000244140625 = 7.32421875 GBto see how long it would take to download will depend on your max download speed.
example 100Mb (Megabit) internet connection
100 Mb / 8 = 12.5 MB ( Megabytes per second )
12.5 MB /1000 = 0.0125 GB (GigaBytes per second)
7.324 GB / 0.0125 GB = 585.92 seconds = 9.72 Min
-
@iKnow0 Thought I had got the noughts wrong.... Bottom line is that with 30TB of plots you are really going to need a 1000Mb/s Connection and even then it will take about a minute to read.
Rich
-
@RichBC I also assumed there would be no network congestion, peak internet hours would also slow it down for sure. A VM in the same location as the storage would work also, but I doubt if it would be cost efficient, at least not yet anyway.
-
@iKnow0
Thanks for clearing that up.
Looks like I might try things when we get near the 10TB mark. See how it fares (fairs?)
-
@iKnow0 So does that means I would be using 7.5 GB of bandwidth for every block? :0 (for a 30TB plot)
-
@Glen yep
-
@iKnow0
Thanks for the math. Looks like it is spot on!
I have a 12MB/s connection and a 2.9TB plot and it takes about 62 secs to read so not far off the figure you quoted if extrapolating out to 30TB.
I think that unless you have a Gbps connection (Who has one of those?) then the plots are probably going to have to be no more than around 10TB max as you probably wont get to finish reading them when the next block appears.
-
Any new experiences with the mining on ACD?
-
Hey guys - do you have any experience with actual minig process on ACD?
I have massive ACD account with ~13TB of plots. I'm mining with my EC2 instance, but unfortunatelly I'm observing very low performance (0.5-2 MB/s) that is unacceptable.
What environment / programs for ACD mounting have you been using during mining?
I've tried with acd_cli on Linux, and NetDrive2 on Windows (on Microsoft Azure instances), but performance on all of them is similar.
-
@Axadiw I am mining under windows on an ec2 t2.micro free account.i am using net drive and am getting anything from 2 to 10MB/shut on a 4TB set of plots.
The problem is that the network performance on this free instances "Low to moderate" according to AWS. This in practise means around 70Mbps but fluctuates depending on how much you are downloading. So as your plot size increases so does the amount your downloading which then proportionally slows the network performance.
I did try the same thing on a Premium VPS service (without a network performance penalty)but found that I got similar performance so not sure what is going on there and can only suspect that the speed may be throttled from Amazon Cloud?
-
Hmm, is it performance is decreased because of that I've been plotting with very small plot files (3-10GB files, together it will be ~3000 separate files) ?
Is it possible to increase parallel reading in miner somehow?
Or maybe is it possible to merge smaller plotfiles somehow?
-
@Axadiw
Having smaller plot files will affect things somewhat. My plots are 24GB in size as larger files fail to upload consistently.
If there was some way to merge them that would be great but I don't think any tool exists? Also if it did, I am assuming that the data would need to be downloaded,joined and then uploaded as a larger file which wouldn't work
-
hmmm, you're right.
BTW. in which region are you hosting your ec2 instance?
-
@Axadiw
eu-west-1c
I wonder if you have had your account locked yet?
That amount of data over that connection speed would mean that you are pretty much downloading constantly.
Have Amazon locked your account as a result of this?
I have had it locked several times now
Easiest way to unlock it is by using their CS chat.
-
I have really enjoyed reading through this project! I have a question though @RichBC in your batch file you list Optimiser where can I find that?
-
@gersey https://forums.burst-team.us/topic/26/plot-optimizer-v1-6-with-gui-1-0-3-j6jq-win
If you go up a couple of levels from this link you will find all the Burst related Software.
Rich
-
@RichBC Thanks for the help. Have you started to mine yet or are you still plotting?
-
@gersey I will be plotting until the end of next week. Then the hire of the VPS comes to an end and I am going to start mining experiments in earnest, without the risk that Uploading and downloading at the same time causes. Currently at about 30TB
Still big risks on can you read fast enough, will the read error problems stop the miner, will Amazon shut it down etc. :-)
Rich
