Plotting and Mining on Amazon Cloud Drive
-
@RichBC sounds good because I think its less than half of the time using the ACD and this does not trigger the shutdown for continuous operation. I'll try the same now; thx!
-
@burster So have you been shutdown just uploading, or have you always been doing a combination of Upload & Mining?
An idea I have is to mine while plotting and then stop mining and upload, all from one location?
Rich
-
Wish I had found this post earlie.
As I have gone down this exact route before finding this valuable info.
I too suffer from a relatively slow broadband speed at home and also didn't want to leave my machine running constantly.
I am using a free ec2 instance to do the mining as the network speed is better than mine at home with the added bonus that it can be left running 24/7.
I too have decided to use a better specked ec2 instance to do the plotting and uploading.
If the ec2 instance is stopped then the charges are minimal when not in use. The bandwidth costs do put a bit of a dent in things though.
I am plotting 16Gb plots as theses have been the most constant when uploading to add.
I am afraid that I am not as organised as Rich and have been doing these manually without a script.
I have been using synbackpro to upload which seems to be doing a reasonable job
So far I have uploaded around 500Gb.
My concern is that the miner is taking increasingly longer to to read the plots and I am wondering how this will pan out as the number of plots get larger.
-
@machasm Great to find someone else treading a similar path... A few comments.
I would be interested in the spec / price of the EC2 instance you have settled on for Plotting and uploading and how long it takes to upload a TB? My testing which was no way exhaustive did not find it that cost effective particularly with the data transfer fees compared to the VPS I have settled on.
That said I have ended up spending much more on the VPS than I originally expected so probably should revisit a paid for EC2. Thing I like about the VPS is that you pay by the Month and knpow exactly how much it is going to cost and there are no additional transfer fees.
Have not seen synbackpro, how reliable do you find it and how well does it cope with the delay between the upload completing and Amazon recognising the file is there? Although it will have a slightly easier task with 16Gb than 25GB.
My longer term concerns remain the same, namely how to get a decent download speed when Mining and will Amazon shut us down?
I will post up my Plot / Optimise / Upload batch file if anyone is interested, although it is no masterpiece. I now have about 21TB uploaded.
Rich
-
@RichBC only had the ec2 instance (m4 xlarge) for a day and not sure what the charges are going to be until a couple of days.
The upload speed is good though at around 500mbps
I think syncbackpro is having difficulty with the delay as well actually and probably explains why it has tried to upload said files twice on some occasions.I need to dig around in the settings to see if there is a delay that can be added before checking if it needs to upload the file again.
I wouldn't mind that script if you have it though.
I will try and update here on progress (or lack of) in the next few days.
-
@machasm I would definitely give rclone a try for the upload. it's working well for me and there is a Delay you will see in my batch file that determines how long it waits for the file to be there before uploading again.
here's the Batch file. it's set up for what works for me. uses wplotgenerator to produce 3 x 25GB files, then optimise, then uploads the 3 file simultaneously to max out the bandwidth using rclone, deletes the files and repeats 80 Times for 2TB.
Not commented, but relatively self explanatory. it uses a 2 Digit prefix (Group) for the plot number because of the size of Digit that the Batch file can cope with.
Also uses a program called Timer.exe to facilitate logging the elapsed time for each stage. Example output I think earlier in this thread.
::Combined Plotter - Optimiser - ACD Uploader. ::Assumes wplotgenerator, Optimiser, rclone & timer.exe are in same Directory as this Batch File. ::Edit the set parameters to suit @echo off set account=14150096290043664893 set/a group=82 set/a nonce=1000000 set/a stagger=102400 set size=6400 set cores=6 set OptMem=1g set transfers=3 set delay=0m30s set ACDPath=ACD:Database\New set/a plot=10 set/a maxplot=90 :Begin @echo %date% Start nonce %group%%nonce% Stagger-%stagger% Size-%size% > Plot-Log-%group%.txt :start timer /nologo /q echo %time% Start Plot %plot% >> Plot-Log-%group%.txt wplotgenerator %account% %group%%nonce% %stagger% %size% %cores% set/a nonce=nonce+stagger set/a plot=plot+1 timer /nologo /n /s >> Plot-Log-%group%.txt timer /nologo /q echo %time% Start Plot %plot% >> Plot-Log-%group%.txt wplotgenerator %account% %group%%nonce% %stagger% %size% %cores% set/a nonce=nonce+stagger set/a plot=plot+1 timer /nologo /n /s >>plot-Log-%group%.txt timer /nologo /q echo %time% Start Plot %plot% >> Plot-Log-%group%.txt wplotgenerator %account% %group%%nonce% %stagger% %size% %cores% set/a nonce=nonce+stagger set/a plot=plot+1 timer /nologo /n /s >> Plot-Log-%group%.txt timer /nologo /q echo %time% Start Optimise >> Plot-Log-%group%.txt Optimizer 30 plots -del 1 -m %OptMem% plots\* timer /nologo /n /s >> Plot-Log-%group%.txt timer /nologo /q echo %time% Start Upload >> Plot-Log-%group%.txt rclone -v --transfers %transfers% --acd-upload-wait-per-gb %delay% copy plots %ACDPath% timer /nologo /n /s >> Plot-Log-%group%.txt Del/q plots\*.* @echo %time% Finish @echo: >> Plot-Log-%group%.txt IF %plot% LSS %maxplot% goto :Start set/a plot=10 set/a group=group+1 set/a nonce=1000000 goto :beginRich
-
@RichBC Thanks for that Rich.
Just experimenting now with each section of the batch file. Trying out rclone first.
Definately seems a far better solution than syncbackpro as the first file I uploaded went without a hitch first time so thanks for that.
Will report on further progress asap.
BTW meant to ask, who are you using your VPS service with?
-
@machasm Glad rclone is working for you, little bit of messing about to get it going but definitely the best I have found so far.
If anything does no make sense in the Batch File just ask. It was specific to what I wanted to do rather than a generic solution.
VPS is with worldservers.com Started with an offer from ebay
Which was £5.99 a Month, however after upping the spec to a minimum sensible spec for plotting etc it ended up at £23. I am in my second Month and will probably call a halt on uploading then and concentrate on getting Mining sorted
Rich
-
@RichBC
Thanks again Rich you have been a massive help
May I ask if you have any charges for bandwidth use with your PVS service?
If so what are they?
-
@machasm Bandwidth usage is unlimited. This was the main reason I moved away from EC2.
Rich
-
@RichBC
Fab
Edit. just visited worldservers.com website and it is for sale?
-
-
@RichBC
Do you have a link to the timere.exe file?
I have all the other files thanks.
-
@machasm Probably other ways the time measurement could be done, but this was the first that I found that worked. handy to keep tabs on how things are going, particularly when you have left it overnight and there are multiple tries to get the files upload.
http://www.gammadyne.com/timer.exe
Rich
-
@RichBC
Got the script running on my VPS.
I have purchased a package from the same guys as you since the network charges of my ec2 instance was killing me.
The performance of the worldservers vps is very slow compared to what I was running.
Doesn't really matter though since it can be left running and uploading plots 24/7.
Still, trying to fault find can take an age!
Also, although they say you have a 1Gbps connection, the throughput is nowhere near that.
A bit of clever advertising I think. I suspect that the hardware that they have specced is right but your virtual machine is only using a fraction of those resources (including network and CPU).
Anyway, I will update on progress and thanks again for your help Rich.
BTW what is your network throughput like Rich? I am beginning to think that they might have given me the wrong spec?
-
@machasm The best I see is around 35MB/s but it's quite variable. I think time of Day but have not tried to track, but as you say not as good as I would have hoped. But at least you know what it's going to cost.
Yes took me sveral days to get things sorted, but the batch file has really helped as it just plows on regardless. What spec of system have you gone for?
Rich
-
@RichBC
Hi Rich.
I went for a 8Gb Ram 450 GB drive but my network throughput right now is diabolical at around 3MB/s! Are you sure you are not talking Mbps? At my current speed it is going to take around 3-4 hours to upload 32Gb.
I have complained and they said that they have added another core and increased RAM to 16Gb.
I can see that they have indeed increased the RAM but the number of cores remains the same at 2.
Also it would appear that the core frequency seems to be stuck at 1Ghz instead of the advertised 3.6Ghz.
Can you verify how many cores and processor speed you have?Edit: when i was running the Ec2 instance I was uploading a 16Gb file in around 10 mins!
-
@machasm I have 6 Cores, 5GB RAM and 150GB of Hard Disk, Processor shows as a Common KVA processor 2.40GHz.
Current speed is 29MB/s and Today it has varied between 24MB/s and 34MB/s. and yes that is Mega Bytes. Here's the log file for the last 4 sets of 3 x 25GB files.
15-Feb-17 Start nonce 851000000 Stagger-102400 Size-6400 11:48:40.86 Start Plot 10 0:22:14.85 12:10:55.74 Start Plot 11 0:22:47.46 12:33:43.25 Start Plot 12 0:23:36.34 12:57:19.66 Start Optimise 1:01:58.06 13:59:17.99 Start Upload 0:53:31.81 14:52:50.84 Start Plot 13 0:23:26.56 15:16:17.45 Start Plot 14 0:24:13.01 15:40:30.51 Start Plot 15 0:22:22.90 16:02:53.54 Start Optimise 0:47:34.95 16:50:28.70 Start Upload 0:41:06.79 17:31:36.36 Start Plot 16 0:20:23.26 17:51:59.66 Start Plot 17 0:19:46.36 18:11:46.03 Start Plot 18 0:20:00.25 18:31:46.39 Start Optimise 0:50:23.29 19:22:09.84 Start Upload 0:43:19.62 20:05:30.18 Start Plot 19 0:19:22.51 20:24:52.70 Start Plot 20 0:19:50.15 20:44:42.88 Start Plot 21 0:19:41.14 21:04:24.32 Start Optimise 1:02:36.84 22:07:01.23 Start Upload 0:48:03.62So between 41 & 53 Minutes to upload each 75GB of Plots. I did find that you needed at least 2 files being uploaded simultaneously to use the bandwidth.
Rich
-
@RichBC Hmm The lack of cores I can live with but the uplink speed is just too slow.
I will see if they can change the spec tomorrow.edit:
How many cores show up when you bring up the task manager?
My processor says common KVM processor 3.6Ghz. Feels like 1Ghz to me though.
Around 2300 Nonces/min during the plot creation which is around 6X slower than my machine at home (4.5Ghz Skylake)
-
@machasm I did not think there was an option on the Uplink speed, not something I discussed. I thought all the packages were specked at 1Gbps? Feel free to refer to me if it helps. I show 6 cores in Task manager. 5352 nonces / minute
Rich
