Plotting and Mining on Amazon Cloud Drive
-
@RichBC Hi, I would like to know, how is your experience with your ACD getting shut down by Amazon so far. Is it the plotting from a private VPS to ACD or is it the mining with an EC2 instance from an ACD without interruptions (around the clock)? I got the answer that the reason for my latest shutdown was "continuous operation" ...
Did you get any problems with them recently, if you did only plotting as you described here?
-
@burster I have had the account locked 4 Times now.
Each time it has only been when I have been Mining / reading plots, which is why at the moment I am just uploading files, both to get more TB and to confirm that it is Mining that is the problem.
The first 3 times they locked the account it was immediately unlocked on a Phone call with no questions. 4th time they said they would investigate and call me back. They did not so I called them and after going through a password reset ( which did nothing) and more on hold they unlocked it.
I questioned what was going on and they kept reminding me that it was just for private use and not for sharing etc.
My conclusion is that the problem is that I was mining from one address, either Home of the Amazon EC2 instance at the same time as uploading, either from home or the VPS that I am renting to give me better upload bandwidth.
My next test will be to stop the uploading and just Mine and see what happens then. I have the VPS rented till the end of this Month and will then give this a go. Both from Home which is way too slow, and then from the EC2 instance.
So bottom line is only theories at the moment, and more experimenting to do. But not ready to give up yet. :-)
All that said your Continuous operation makes a lot of sense, and I was far from certain that the operator I was speaking to had any idea why it had been locked.
Rich
-
@RichBC thx so far, so your continuously uploading did not trigger a shutdown yet? Since when do you upload? The guy told me also, if I do not change my behavior we'll have to talk every three days...
-
@burster I have been continuously uploading for over a Month now, probably around 35MB/s However it is not continuous because I have to create the plot on the VPS first.
In order to save money I have a bit of a minimalist spec and this means that the sequence is about 1 Hour 50 Mins plotting and optimising 3 x 25GB files and then anything between 40 Minutes and an Hour, depending on the VPS bandwidth simultaneously uploading the 3 files.
Rich
-
@RichBC sounds good because I think its less than half of the time using the ACD and this does not trigger the shutdown for continuous operation. I'll try the same now; thx!
-
@burster So have you been shutdown just uploading, or have you always been doing a combination of Upload & Mining?
An idea I have is to mine while plotting and then stop mining and upload, all from one location?
Rich
-
Wish I had found this post earlie.
As I have gone down this exact route before finding this valuable info.
I too suffer from a relatively slow broadband speed at home and also didn't want to leave my machine running constantly.
I am using a free ec2 instance to do the mining as the network speed is better than mine at home with the added bonus that it can be left running 24/7.
I too have decided to use a better specked ec2 instance to do the plotting and uploading.
If the ec2 instance is stopped then the charges are minimal when not in use. The bandwidth costs do put a bit of a dent in things though.
I am plotting 16Gb plots as theses have been the most constant when uploading to add.
I am afraid that I am not as organised as Rich and have been doing these manually without a script.
I have been using synbackpro to upload which seems to be doing a reasonable job
So far I have uploaded around 500Gb.
My concern is that the miner is taking increasingly longer to to read the plots and I am wondering how this will pan out as the number of plots get larger.
-
@machasm Great to find someone else treading a similar path... A few comments.
I would be interested in the spec / price of the EC2 instance you have settled on for Plotting and uploading and how long it takes to upload a TB? My testing which was no way exhaustive did not find it that cost effective particularly with the data transfer fees compared to the VPS I have settled on.
That said I have ended up spending much more on the VPS than I originally expected so probably should revisit a paid for EC2. Thing I like about the VPS is that you pay by the Month and knpow exactly how much it is going to cost and there are no additional transfer fees.
Have not seen synbackpro, how reliable do you find it and how well does it cope with the delay between the upload completing and Amazon recognising the file is there? Although it will have a slightly easier task with 16Gb than 25GB.
My longer term concerns remain the same, namely how to get a decent download speed when Mining and will Amazon shut us down?
I will post up my Plot / Optimise / Upload batch file if anyone is interested, although it is no masterpiece. I now have about 21TB uploaded.
Rich
-
@RichBC only had the ec2 instance (m4 xlarge) for a day and not sure what the charges are going to be until a couple of days.
The upload speed is good though at around 500mbps
I think syncbackpro is having difficulty with the delay as well actually and probably explains why it has tried to upload said files twice on some occasions.I need to dig around in the settings to see if there is a delay that can be added before checking if it needs to upload the file again.
I wouldn't mind that script if you have it though.
I will try and update here on progress (or lack of) in the next few days.
-
@machasm I would definitely give rclone a try for the upload. it's working well for me and there is a Delay you will see in my batch file that determines how long it waits for the file to be there before uploading again.
here's the Batch file. it's set up for what works for me. uses wplotgenerator to produce 3 x 25GB files, then optimise, then uploads the 3 file simultaneously to max out the bandwidth using rclone, deletes the files and repeats 80 Times for 2TB.
Not commented, but relatively self explanatory. it uses a 2 Digit prefix (Group) for the plot number because of the size of Digit that the Batch file can cope with.
Also uses a program called Timer.exe to facilitate logging the elapsed time for each stage. Example output I think earlier in this thread.
::Combined Plotter - Optimiser - ACD Uploader. ::Assumes wplotgenerator, Optimiser, rclone & timer.exe are in same Directory as this Batch File. ::Edit the set parameters to suit @echo off set account=14150096290043664893 set/a group=82 set/a nonce=1000000 set/a stagger=102400 set size=6400 set cores=6 set OptMem=1g set transfers=3 set delay=0m30s set ACDPath=ACD:Database\New set/a plot=10 set/a maxplot=90 :Begin @echo %date% Start nonce %group%%nonce% Stagger-%stagger% Size-%size% > Plot-Log-%group%.txt :start timer /nologo /q echo %time% Start Plot %plot% >> Plot-Log-%group%.txt wplotgenerator %account% %group%%nonce% %stagger% %size% %cores% set/a nonce=nonce+stagger set/a plot=plot+1 timer /nologo /n /s >> Plot-Log-%group%.txt timer /nologo /q echo %time% Start Plot %plot% >> Plot-Log-%group%.txt wplotgenerator %account% %group%%nonce% %stagger% %size% %cores% set/a nonce=nonce+stagger set/a plot=plot+1 timer /nologo /n /s >>plot-Log-%group%.txt timer /nologo /q echo %time% Start Plot %plot% >> Plot-Log-%group%.txt wplotgenerator %account% %group%%nonce% %stagger% %size% %cores% set/a nonce=nonce+stagger set/a plot=plot+1 timer /nologo /n /s >> Plot-Log-%group%.txt timer /nologo /q echo %time% Start Optimise >> Plot-Log-%group%.txt Optimizer 30 plots -del 1 -m %OptMem% plots\* timer /nologo /n /s >> Plot-Log-%group%.txt timer /nologo /q echo %time% Start Upload >> Plot-Log-%group%.txt rclone -v --transfers %transfers% --acd-upload-wait-per-gb %delay% copy plots %ACDPath% timer /nologo /n /s >> Plot-Log-%group%.txt Del/q plots\*.* @echo %time% Finish @echo: >> Plot-Log-%group%.txt IF %plot% LSS %maxplot% goto :Start set/a plot=10 set/a group=group+1 set/a nonce=1000000 goto :beginRich
-
@RichBC Thanks for that Rich.
Just experimenting now with each section of the batch file. Trying out rclone first.
Definately seems a far better solution than syncbackpro as the first file I uploaded went without a hitch first time so thanks for that.
Will report on further progress asap.
BTW meant to ask, who are you using your VPS service with?
-
@machasm Glad rclone is working for you, little bit of messing about to get it going but definitely the best I have found so far.
If anything does no make sense in the Batch File just ask. It was specific to what I wanted to do rather than a generic solution.
VPS is with worldservers.com Started with an offer from ebay
Which was £5.99 a Month, however after upping the spec to a minimum sensible spec for plotting etc it ended up at £23. I am in my second Month and will probably call a halt on uploading then and concentrate on getting Mining sorted
Rich
-
@RichBC
Thanks again Rich you have been a massive help
May I ask if you have any charges for bandwidth use with your PVS service?
If so what are they?
-
@machasm Bandwidth usage is unlimited. This was the main reason I moved away from EC2.
Rich
-
@RichBC
Fab
Edit. just visited worldservers.com website and it is for sale?
-
-
@RichBC
Do you have a link to the timere.exe file?
I have all the other files thanks.
-
@machasm Probably other ways the time measurement could be done, but this was the first that I found that worked. handy to keep tabs on how things are going, particularly when you have left it overnight and there are multiple tries to get the files upload.
http://www.gammadyne.com/timer.exe
Rich
-
@RichBC
Got the script running on my VPS.
I have purchased a package from the same guys as you since the network charges of my ec2 instance was killing me.
The performance of the worldservers vps is very slow compared to what I was running.
Doesn't really matter though since it can be left running and uploading plots 24/7.
Still, trying to fault find can take an age!
Also, although they say you have a 1Gbps connection, the throughput is nowhere near that.
A bit of clever advertising I think. I suspect that the hardware that they have specced is right but your virtual machine is only using a fraction of those resources (including network and CPU).
Anyway, I will update on progress and thanks again for your help Rich.
BTW what is your network throughput like Rich? I am beginning to think that they might have given me the wrong spec?
-
@machasm The best I see is around 35MB/s but it's quite variable. I think time of Day but have not tried to track, but as you say not as good as I would have hoped. But at least you know what it's going to cost.
Yes took me sveral days to get things sorted, but the batch file has really helped as it just plows on regardless. What spec of system have you gone for?
Rich

