Plotting and Mining on Amazon Cloud Drive

[miraj]

Thanks for your thoughts, Rich.

I try to copy a plot file to the ACD\burst\plots folder but it says the folder already exists. That is correct, it does already exist on the cloud drive. I just want to put a new file into the folder. rclone doesn't let me so I had to create a new folder called b1. Then when the file is uploaded, I have to move it to burst\plots. Is this the correct code to copy a file to ACD?

rclone copy C:\burst\upload\3051863391608293710_15220161_102336_102336 acd:burst\plots

(I called my amazon cloud drive - acd)

Thanks!
Mike

[RichBC]

@miraj Strange all of my copies to ACD have been to existing folders without that message?

[miraj]

@RichBC

Got it. Seems the ACD is very case sensitive. My Burst folder has capital "B". Working great now. Just have to wait 8 hours till its finished!

Thanks!
Mike

[RichBC]

@miraj OK great. Also if you include a directory in the ACD path it will create it.

Be careful around 50GB there is max file size limit. I have been ok with 48GB & 48.8GB but have also found, at least on slow transfers, that failures grow significantly above 25GB

Rich

[miraj]

@RichBC Is there code in rclone to upload multiple files, but only upload the next file after the previous one is finished? I'm trying to create a bunch of plot files, then when I'm at work, let this run and it uploads all of the files one at a time. I've noticed that my upload connection drops about once every 3 hours or so. It has to keep restarting the uploads. If that happens, i'd only like to lose the file currently being uploaded.

Thanks,
MIke

[RichBC]

@miraj Yes you need to put --transfers 1 on the command line.

I think the default is 4. From my home connection I use 1, from the VPS I am using 2 or 3 as it then maxes out the Upload bandwidth that is available.

Rich

[miraj]

@RichBC That's exactly what I needed!

Thanks,
Mike

[RichBC]

Quick update on how the Cloud Mining is going.

To speed up the upload process as an experiment I have rented a VPS for a Month. After some playing I have settled on a minimum spec of 6 Cores, 5GB RAM and 150GB of Hard Disk.

I have written a Batch file that plots 3 x 25GB plots then optimises them and uploads the 3 plots simultaneously to max out the available bandwidth. This process enables me to upload 1TB every 2 Days. There is about a 10% failure rate on the uploads. I am now up to 9TB.

So this will work out at about $2.50 / TB which is just about acceptable. I have only been mining the Plots sparingly as I think it might be the Mining process that get's the Drive shut down? I have however been Mining for the last 24 Hours without problem.

The big issue when a decent amount of plots are up will be the read speed. With a Download Bandwidth of 8MB/s the 9TB takes 4 Minutes to read. :-( I have also been testing Mining from an EC2 Free Instance and get at least twice that speed, but still not really fast enough.

On the faster speeds I have also seen another problem which is a "error ReadFiles" Sometimes it just a couple of errors but on other occasions you get a complete never ending string of errors. Need to understand this and find a solution otherwise it could be a showstopper. perhaps @blago might have some ideas?

If I had confidence that Amazon would not shut down the Drive I think this would have the makings of a great low cost, high value Mining Asset as the cost of creating the plots is only $2.50 /TB and the Annual storage costs only an additional $50 for up to 100TB.

Rich

[gpedro]

@RichBC said in Plotting and Mining on Amazon Cloud Drive:

On the faster speeds I have also seen another problem which is a "error ReadFiles" Sometimes it just a couple of errors but on other occasions you get a complete never ending string of errors. Need to understand this and find a solution otherwise it could be a showstopper. perhaps @blago might have some ideas?

That error is due to the internet connection if i am not wrong... I seen it every once in a while while mining with clouds ;D

[RichBC]

@gpedro Yes could be and that is what Blago thought. Difficult for me to tell as the problem comes when I am using the Amazon EC2 free Instance, because their Bandwidth is much better than mine.

Rich

[gpedro]

@RichBC I think it have something to do with lost packages not really bandwidth tbh ;D

[RichBC]

Quick up date on progress. I have been concentrating on Plotting rather than Mining. Currently I am up to 17TB of the 25GB files.

My Batch file now working quite well and I have developed it so that it starts a Group with a 2 Digit prefix to the plot numbers, then plots 80 sequential 25GB files (2TB). It then increments the prefix and does another 2TB of files.

At the moment it uploads them to a directory called new at ACD and I manually put them into folders of 2TB each. The Batch File could easilly do this but it gives me an opportunity to check things are running ok.

The batch file also logs to a text file each of the stages along with time stamps to let me monitor progress. Biggest problem with the uploads is the delay in ACD recognising the file is there and that some files have to be uploaded again which slows things considerably.

However rclone handles this well and the file get there eventually. Heres a typical log file.

14-Jan-17 Start nonce 941000000 Stagger-102400 Size-6400 
 9:34:01.68 Start Plot 0 
0:24:40.04
 9:58:41.76 Start Plot 1 
0:24:41.65
10:23:23.43 Start Plot 2 
0:20:31.82
10:43:55.60 Start Optimise 
0:44:29.81
11:28:25.66 Start Upload 
0:37:37.12
 
12:06:03.58 Start Plot 3 
0:20:14.96
12:26:18.64 Start Plot 4 
0:20:16.98
12:46:35.64 Start Plot 5 
0:19:59.00
13:06:34.66 Start Optimise 
0:41:23.09
13:47:57.86 Start Upload 
0:41:23.71
 
14:29:22.45 Start Plot 6 
0:23:05.12
14:52:27.80 Start Plot 7 
0:23:53.23
15:16:21.28 Start Plot 8 
0:22:48.35
15:39:10.41 Start Optimise 
1:00:40.29
16:39:50.98 Start Upload 
1:53:19.70

I do 3 x 25GB plots, log file shows start time for each, then on the following line how long it took. Then the files are optimised, and then uploaded, the 3 files simultaneously as this maxes out the bandwidth at the VPS. So a good sequence of 3 files, 75GB, takes about 2 1/2 Hours, however you will see a longer time for the last upload, indicating there will have been a failure.

This is from the VPS I am renting, the cost is higher than previously reported due to all the real world problems at both ends of the chain that you hit and I would estimate about $4 / TB but have not redone the Maths.

Rich

[burster]

@RichBC Hi, I would like to know, how is your experience with your ACD getting shut down by Amazon so far. Is it the plotting from a private VPS to ACD or is it the mining with an EC2 instance from an ACD without interruptions (around the clock)? I got the answer that the reason for my latest shutdown was "continuous operation" ...
Did you get any problems with them recently, if you did only plotting as you described here?

[RichBC]

@burster I have had the account locked 4 Times now.

Each time it has only been when I have been Mining / reading plots, which is why at the moment I am just uploading files, both to get more TB and to confirm that it is Mining that is the problem.

The first 3 times they locked the account it was immediately unlocked on a Phone call with no questions. 4th time they said they would investigate and call me back. They did not so I called them and after going through a password reset ( which did nothing) and more on hold they unlocked it.

I questioned what was going on and they kept reminding me that it was just for private use and not for sharing etc.

My conclusion is that the problem is that I was mining from one address, either Home of the Amazon EC2 instance at the same time as uploading, either from home or the VPS that I am renting to give me better upload bandwidth.

My next test will be to stop the uploading and just Mine and see what happens then. I have the VPS rented till the end of this Month and will then give this a go. Both from Home which is way too slow, and then from the EC2 instance.

So bottom line is only theories at the moment, and more experimenting to do. But not ready to give up yet. :-)

All that said your Continuous operation makes a lot of sense, and I was far from certain that the operator I was speaking to had any idea why it had been locked.

Rich

[burster]

@RichBC thx so far, so your continuously uploading did not trigger a shutdown yet? Since when do you upload? The guy told me also, if I do not change my behavior we'll have to talk every three days...

[RichBC]

@burster I have been continuously uploading for over a Month now, probably around 35MB/s However it is not continuous because I have to create the plot on the VPS first.

In order to save money I have a bit of a minimalist spec and this means that the sequence is about 1 Hour 50 Mins plotting and optimising 3 x 25GB files and then anything between 40 Minutes and an Hour, depending on the VPS bandwidth simultaneously uploading the 3 files.

Rich

[burster]

@RichBC sounds good because I think its less than half of the time using the ACD and this does not trigger the shutdown for continuous operation. I'll try the same now; thx!

[RichBC]

@burster So have you been shutdown just uploading, or have you always been doing a combination of Upload & Mining?

An idea I have is to mine while plotting and then stop mining and upload, all from one location?

Rich

[machasm]

Wish I had found this post earlie.
As I have gone down this exact route before finding this valuable info.
I too suffer from a relatively slow broadband speed at home and also didn't want to leave my machine running constantly.
I am using a free ec2 instance to do the mining as the network speed is better than mine at home with the added bonus that it can be left running 24/7.
I too have decided to use a better specked ec2 instance to do the plotting and uploading.
If the ec2 instance is stopped then the charges are minimal when not in use. The bandwidth costs do put a bit of a dent in things though.
I am plotting 16Gb plots as theses have been the most constant when uploading to add.
I am afraid that I am not as organised as Rich and have been doing these manually without a script.
I have been using synbackpro to upload which seems to be doing a reasonable job
So far I have uploaded around 500Gb.
My concern is that the miner is taking increasingly longer to to read the plots and I am wondering how this will pan out as the number of plots get larger.

[RichBC]

@machasm Great to find someone else treading a similar path... A few comments.

I would be interested in the spec / price of the EC2 instance you have settled on for Plotting and uploading and how long it takes to upload a TB? My testing which was no way exhaustive did not find it that cost effective particularly with the data transfer fees compared to the VPS I have settled on.

That said I have ended up spending much more on the VPS than I originally expected so probably should revisit a paid for EC2. Thing I like about the VPS is that you pay by the Month and knpow exactly how much it is going to cost and there are no additional transfer fees.

Have not seen synbackpro, how reliable do you find it and how well does it cope with the delay between the upload completing and Amazon recognising the file is there? Although it will have a slightly easier task with 16Gb than 25GB.

My longer term concerns remain the same, namely how to get a decent download speed when Mining and will Amazon shut us down?

I will post up my Plot / Optimise / Upload batch file if anyone is interested, although it is no masterpiece. I now have about 21TB uploaded.

Rich