Plotting and Mining on Amazon Cloud Drive
-
@RichBC only had the ec2 instance (m4 xlarge) for a day and not sure what the charges are going to be until a couple of days.
The upload speed is good though at around 500mbps
I think syncbackpro is having difficulty with the delay as well actually and probably explains why it has tried to upload said files twice on some occasions.I need to dig around in the settings to see if there is a delay that can be added before checking if it needs to upload the file again.
I wouldn't mind that script if you have it though.
I will try and update here on progress (or lack of) in the next few days.
-
@machasm I would definitely give rclone a try for the upload. it's working well for me and there is a Delay you will see in my batch file that determines how long it waits for the file to be there before uploading again.
here's the Batch file. it's set up for what works for me. uses wplotgenerator to produce 3 x 25GB files, then optimise, then uploads the 3 file simultaneously to max out the bandwidth using rclone, deletes the files and repeats 80 Times for 2TB.
Not commented, but relatively self explanatory. it uses a 2 Digit prefix (Group) for the plot number because of the size of Digit that the Batch file can cope with.
Also uses a program called Timer.exe to facilitate logging the elapsed time for each stage. Example output I think earlier in this thread.
::Combined Plotter - Optimiser - ACD Uploader. ::Assumes wplotgenerator, Optimiser, rclone & timer.exe are in same Directory as this Batch File. ::Edit the set parameters to suit @echo off set account=14150096290043664893 set/a group=82 set/a nonce=1000000 set/a stagger=102400 set size=6400 set cores=6 set OptMem=1g set transfers=3 set delay=0m30s set ACDPath=ACD:Database\New set/a plot=10 set/a maxplot=90 :Begin @echo %date% Start nonce %group%%nonce% Stagger-%stagger% Size-%size% > Plot-Log-%group%.txt :start timer /nologo /q echo %time% Start Plot %plot% >> Plot-Log-%group%.txt wplotgenerator %account% %group%%nonce% %stagger% %size% %cores% set/a nonce=nonce+stagger set/a plot=plot+1 timer /nologo /n /s >> Plot-Log-%group%.txt timer /nologo /q echo %time% Start Plot %plot% >> Plot-Log-%group%.txt wplotgenerator %account% %group%%nonce% %stagger% %size% %cores% set/a nonce=nonce+stagger set/a plot=plot+1 timer /nologo /n /s >>plot-Log-%group%.txt timer /nologo /q echo %time% Start Plot %plot% >> Plot-Log-%group%.txt wplotgenerator %account% %group%%nonce% %stagger% %size% %cores% set/a nonce=nonce+stagger set/a plot=plot+1 timer /nologo /n /s >> Plot-Log-%group%.txt timer /nologo /q echo %time% Start Optimise >> Plot-Log-%group%.txt Optimizer 30 plots -del 1 -m %OptMem% plots\* timer /nologo /n /s >> Plot-Log-%group%.txt timer /nologo /q echo %time% Start Upload >> Plot-Log-%group%.txt rclone -v --transfers %transfers% --acd-upload-wait-per-gb %delay% copy plots %ACDPath% timer /nologo /n /s >> Plot-Log-%group%.txt Del/q plots\*.* @echo %time% Finish @echo: >> Plot-Log-%group%.txt IF %plot% LSS %maxplot% goto :Start set/a plot=10 set/a group=group+1 set/a nonce=1000000 goto :beginRich
-
@RichBC Thanks for that Rich.
Just experimenting now with each section of the batch file. Trying out rclone first.
Definately seems a far better solution than syncbackpro as the first file I uploaded went without a hitch first time so thanks for that.
Will report on further progress asap.
BTW meant to ask, who are you using your VPS service with?
-
@machasm Glad rclone is working for you, little bit of messing about to get it going but definitely the best I have found so far.
If anything does no make sense in the Batch File just ask. It was specific to what I wanted to do rather than a generic solution.
VPS is with worldservers.com Started with an offer from ebay
Which was £5.99 a Month, however after upping the spec to a minimum sensible spec for plotting etc it ended up at £23. I am in my second Month and will probably call a halt on uploading then and concentrate on getting Mining sorted
Rich
-
@RichBC
Thanks again Rich you have been a massive help
May I ask if you have any charges for bandwidth use with your PVS service?
If so what are they?
-
@machasm Bandwidth usage is unlimited. This was the main reason I moved away from EC2.
Rich
-
@RichBC
Fab
Edit. just visited worldservers.com website and it is for sale?
-
-
@RichBC
Do you have a link to the timere.exe file?
I have all the other files thanks.
-
@machasm Probably other ways the time measurement could be done, but this was the first that I found that worked. handy to keep tabs on how things are going, particularly when you have left it overnight and there are multiple tries to get the files upload.
http://www.gammadyne.com/timer.exe
Rich
-
@RichBC
Got the script running on my VPS.
I have purchased a package from the same guys as you since the network charges of my ec2 instance was killing me.
The performance of the worldservers vps is very slow compared to what I was running.
Doesn't really matter though since it can be left running and uploading plots 24/7.
Still, trying to fault find can take an age!
Also, although they say you have a 1Gbps connection, the throughput is nowhere near that.
A bit of clever advertising I think. I suspect that the hardware that they have specced is right but your virtual machine is only using a fraction of those resources (including network and CPU).
Anyway, I will update on progress and thanks again for your help Rich.
BTW what is your network throughput like Rich? I am beginning to think that they might have given me the wrong spec?
-
@machasm The best I see is around 35MB/s but it's quite variable. I think time of Day but have not tried to track, but as you say not as good as I would have hoped. But at least you know what it's going to cost.
Yes took me sveral days to get things sorted, but the batch file has really helped as it just plows on regardless. What spec of system have you gone for?
Rich
-
@RichBC
Hi Rich.
I went for a 8Gb Ram 450 GB drive but my network throughput right now is diabolical at around 3MB/s! Are you sure you are not talking Mbps? At my current speed it is going to take around 3-4 hours to upload 32Gb.
I have complained and they said that they have added another core and increased RAM to 16Gb.
I can see that they have indeed increased the RAM but the number of cores remains the same at 2.
Also it would appear that the core frequency seems to be stuck at 1Ghz instead of the advertised 3.6Ghz.
Can you verify how many cores and processor speed you have?Edit: when i was running the Ec2 instance I was uploading a 16Gb file in around 10 mins!
-
@machasm I have 6 Cores, 5GB RAM and 150GB of Hard Disk, Processor shows as a Common KVA processor 2.40GHz.
Current speed is 29MB/s and Today it has varied between 24MB/s and 34MB/s. and yes that is Mega Bytes. Here's the log file for the last 4 sets of 3 x 25GB files.
15-Feb-17 Start nonce 851000000 Stagger-102400 Size-6400 11:48:40.86 Start Plot 10 0:22:14.85 12:10:55.74 Start Plot 11 0:22:47.46 12:33:43.25 Start Plot 12 0:23:36.34 12:57:19.66 Start Optimise 1:01:58.06 13:59:17.99 Start Upload 0:53:31.81 14:52:50.84 Start Plot 13 0:23:26.56 15:16:17.45 Start Plot 14 0:24:13.01 15:40:30.51 Start Plot 15 0:22:22.90 16:02:53.54 Start Optimise 0:47:34.95 16:50:28.70 Start Upload 0:41:06.79 17:31:36.36 Start Plot 16 0:20:23.26 17:51:59.66 Start Plot 17 0:19:46.36 18:11:46.03 Start Plot 18 0:20:00.25 18:31:46.39 Start Optimise 0:50:23.29 19:22:09.84 Start Upload 0:43:19.62 20:05:30.18 Start Plot 19 0:19:22.51 20:24:52.70 Start Plot 20 0:19:50.15 20:44:42.88 Start Plot 21 0:19:41.14 21:04:24.32 Start Optimise 1:02:36.84 22:07:01.23 Start Upload 0:48:03.62So between 41 & 53 Minutes to upload each 75GB of Plots. I did find that you needed at least 2 files being uploaded simultaneously to use the bandwidth.
Rich
-
@RichBC Hmm The lack of cores I can live with but the uplink speed is just too slow.
I will see if they can change the spec tomorrow.edit:
How many cores show up when you bring up the task manager?
My processor says common KVM processor 3.6Ghz. Feels like 1Ghz to me though.
Around 2300 Nonces/min during the plot creation which is around 6X slower than my machine at home (4.5Ghz Skylake)
-
@machasm I did not think there was an option on the Uplink speed, not something I discussed. I thought all the packages were specked at 1Gbps? Feel free to refer to me if it helps. I show 6 cores in Task manager. 5352 nonces / minute
Rich
-
@RichBC
I have sent you a chat for PM Rich
-
@RichBC yes, the shutdown resulted from uploading and mining simultaneously. It engaged the acd most of the time. Your idea of simultaneous plotting and mining from one location sounds brilliant! I think the alarm is triggered when reading more than xx% of the time..
I am uploading now since one week with about 50% active, 50% quiet.
-
@burster
I had my account locked once so far but I was accessing the drive from multiple locations.
When you had your account locked were you uploading and mining from the same machine or were you mining from a different machine?
I am trying to work out if it is the simultaneous upload and download or if it is locked due to accessing from different IPs.
-
@machasm
Answered my own question. Managed to get locked out again last night.
I was uploading from one machine but then noticed I had two plots that were not optimised so I downloaded them, optimised them then uploaded them again.
It was shortly after the downloads that my account got locked all on the same machine so single IP.
I was careful to ensure that I was logged out of ACD from all other instances so had to be that that caused the account lock trigger.
