Plotting and Mining on Amazon Cloud Drive



  • @machasm Great to find someone else treading a similar path... A few comments.

    I would be interested in the spec / price of the EC2 instance you have settled on for Plotting and uploading and how long it takes to upload a TB? My testing which was no way exhaustive did not find it that cost effective particularly with the data transfer fees compared to the VPS I have settled on.

    That said I have ended up spending much more on the VPS than I originally expected so probably should revisit a paid for EC2. Thing I like about the VPS is that you pay by the Month and knpow exactly how much it is going to cost and there are no additional transfer fees.

    Have not seen synbackpro, how reliable do you find it and how well does it cope with the delay between the upload completing and Amazon recognising the file is there? Although it will have a slightly easier task with 16Gb than 25GB.

    My longer term concerns remain the same, namely how to get a decent download speed when Mining and will Amazon shut us down?

    I will post up my Plot / Optimise / Upload batch file if anyone is interested, although it is no masterpiece. I now have about 21TB uploaded.

    Rich



  • @RichBC only had the ec2 instance (m4 xlarge) for a day and not sure what the charges are going to be until a couple of days.
    The upload speed is good though at around 500mbps
    I think syncbackpro is having difficulty with the delay as well actually and probably explains why it has tried to upload said files twice on some occasions.

    I need to dig around in the settings to see if there is a delay that can be added before checking if it needs to upload the file again.
    I wouldn't mind that script if you have it though.
    I will try and update here on progress (or lack of) in the next few days.



  • @machasm I would definitely give rclone a try for the upload. it's working well for me and there is a Delay you will see in my batch file that determines how long it waits for the file to be there before uploading again.

    here's the Batch file. it's set up for what works for me. uses wplotgenerator to produce 3 x 25GB files, then optimise, then uploads the 3 file simultaneously to max out the bandwidth using rclone, deletes the files and repeats 80 Times for 2TB.

    Not commented, but relatively self explanatory. it uses a 2 Digit prefix (Group) for the plot number because of the size of Digit that the Batch file can cope with.

    Also uses a program called Timer.exe to facilitate logging the elapsed time for each stage. Example output I think earlier in this thread.

    ::Combined Plotter - Optimiser - ACD Uploader.
    ::Assumes wplotgenerator, Optimiser,  rclone & timer.exe are in same Directory as this Batch File.
    ::Edit the set parameters to suit
    
    @echo off
    
    set account=14150096290043664893
    set/a group=82
    set/a nonce=1000000
    set/a stagger=102400
    set size=6400
    set cores=6
    set OptMem=1g
    set transfers=3
    set delay=0m30s
    set ACDPath=ACD:Database\New
    set/a plot=10
    set/a maxplot=90
    :Begin
    @echo %date% Start nonce %group%%nonce% Stagger-%stagger% Size-%size% > Plot-Log-%group%.txt
    :start
    
    timer /nologo /q
    echo %time% Start Plot %plot% >> Plot-Log-%group%.txt
    
    wplotgenerator %account% %group%%nonce% %stagger% %size% %cores%
    set/a nonce=nonce+stagger
    set/a plot=plot+1
    timer /nologo /n /s >> Plot-Log-%group%.txt
    timer /nologo /q
    echo %time% Start Plot %plot% >> Plot-Log-%group%.txt
    
    wplotgenerator %account% %group%%nonce% %stagger% %size% %cores%
    set/a nonce=nonce+stagger
    set/a plot=plot+1
    timer /nologo /n /s >>plot-Log-%group%.txt
    
    timer /nologo /q
    echo %time% Start Plot %plot% >> Plot-Log-%group%.txt
    
    wplotgenerator %account% %group%%nonce% %stagger% %size% %cores%
    set/a nonce=nonce+stagger
    set/a plot=plot+1
    timer /nologo /n /s >> Plot-Log-%group%.txt
    
    timer /nologo /q
    echo %time% Start Optimise >> Plot-Log-%group%.txt
    
    Optimizer 30 plots -del 1 -m %OptMem% plots\*
    timer /nologo /n /s >> Plot-Log-%group%.txt
    
    timer /nologo /q
    echo %time% Start Upload >> Plot-Log-%group%.txt
    
    rclone -v --transfers %transfers% --acd-upload-wait-per-gb %delay% copy 
    
    plots %ACDPath%
    timer /nologo /n /s >> Plot-Log-%group%.txt
    
    Del/q plots\*.*
    @echo %time% Finish
    @echo: >> Plot-Log-%group%.txt
    
    IF %plot% LSS %maxplot% goto :Start
    set/a plot=10
    set/a group=group+1
    set/a nonce=1000000
    goto :begin
    

    Rich



  • @RichBC Thanks for that Rich.
    Just experimenting now with each section of the batch file. Trying out rclone first.
    Definately seems a far better solution than syncbackpro as the first file I uploaded went without a hitch first time so thanks for that.
    Will report on further progress asap.
    BTW meant to ask, who are you using your VPS service with?



  • @machasm Glad rclone is working for you, little bit of messing about to get it going but definitely the best I have found so far.

    If anything does no make sense in the Batch File just ask. It was specific to what I wanted to do rather than a generic solution.

    VPS is with worldservers.com Started with an offer from ebay

    http://www.ebay.co.uk/itm/322368512429?_trksid=p2060353.m2749.l2649&var=511299944753&ssPageName=STRK%3AMEBIDX%3AIT

    Which was £5.99 a Month, however after upping the spec to a minimum sensible spec for plotting etc it ended up at £23. I am in my second Month and will probably call a halt on uploading then and concentrate on getting Mining sorted

    Rich



  • @RichBC
    Thanks again Rich you have been a massive help
    May I ask if you have any charges for bandwidth use with your PVS service?
    If so what are they?



  • @machasm Bandwidth usage is unlimited. This was the main reason I moved away from EC2.

    Rich



  • @RichBC
    Fab
    Edit. just visited worldservers.com website and it is for sale?



  • @machasm Sorry my mistake.

    theworldservers.com

    Rich



  • @RichBC
    Do you have a link to the timere.exe file?
    I have all the other files thanks.



  • @machasm Probably other ways the time measurement could be done, but this was the first that I found that worked. handy to keep tabs on how things are going, particularly when you have left it overnight and there are multiple tries to get the files upload.

    http://www.gammadyne.com/timer.exe

    Rich



  • @RichBC
    Got the script running on my VPS.
    I have purchased a package from the same guys as you since the network charges of my ec2 instance was killing me.
    The performance of the worldservers vps is very slow compared to what I was running.
    Doesn't really matter though since it can be left running and uploading plots 24/7.
    Still, trying to fault find can take an age!
    Also, although they say you have a 1Gbps connection, the throughput is nowhere near that.
    A bit of clever advertising I think. I suspect that the hardware that they have specced is right but your virtual machine is only using a fraction of those resources (including network and CPU).
    Anyway, I will update on progress and thanks again for your help Rich.
    BTW what is your network throughput like Rich? I am beginning to think that they might have given me the wrong spec?



  • @machasm The best I see is around 35MB/s but it's quite variable. I think time of Day but have not tried to track, but as you say not as good as I would have hoped. But at least you know what it's going to cost.

    Yes took me sveral days to get things sorted, but the batch file has really helped as it just plows on regardless. What spec of system have you gone for?

    Rich



  • @RichBC
    Hi Rich.
    I went for a 8Gb Ram 450 GB drive but my network throughput right now is diabolical at around 3MB/s! Are you sure you are not talking Mbps? At my current speed it is going to take around 3-4 hours to upload 32Gb.
    I have complained and they said that they have added another core and increased RAM to 16Gb.
    I can see that they have indeed increased the RAM but the number of cores remains the same at 2.
    Also it would appear that the core frequency seems to be stuck at 1Ghz instead of the advertised 3.6Ghz.
    Can you verify how many cores and processor speed you have?

    Edit: when i was running the Ec2 instance I was uploading a 16Gb file in around 10 mins!



  • @machasm I have 6 Cores, 5GB RAM and 150GB of Hard Disk, Processor shows as a Common KVA processor 2.40GHz.

    Current speed is 29MB/s and Today it has varied between 24MB/s and 34MB/s. and yes that is Mega Bytes. Here's the log file for the last 4 sets of 3 x 25GB files.

    15-Feb-17 Start nonce 851000000 Stagger-102400 Size-6400 
    11:48:40.86 Start Plot 10 
    0:22:14.85
    12:10:55.74 Start Plot 11 
    0:22:47.46
    12:33:43.25 Start Plot 12 
    0:23:36.34
    12:57:19.66 Start Optimise 
    1:01:58.06
    13:59:17.99 Start Upload 
    0:53:31.81
     
    14:52:50.84 Start Plot 13 
    0:23:26.56
    15:16:17.45 Start Plot 14 
    0:24:13.01
    15:40:30.51 Start Plot 15 
    0:22:22.90
    16:02:53.54 Start Optimise 
    0:47:34.95
    16:50:28.70 Start Upload 
    0:41:06.79
     
    17:31:36.36 Start Plot 16 
    0:20:23.26
    17:51:59.66 Start Plot 17 
    0:19:46.36
    18:11:46.03 Start Plot 18 
    0:20:00.25
    18:31:46.39 Start Optimise 
    0:50:23.29
    19:22:09.84 Start Upload 
    0:43:19.62
     
    20:05:30.18 Start Plot 19 
    0:19:22.51
    20:24:52.70 Start Plot 20 
    0:19:50.15
    20:44:42.88 Start Plot 21 
    0:19:41.14
    21:04:24.32 Start Optimise 
    1:02:36.84
    22:07:01.23 Start Upload 
    0:48:03.62
    

    So between 41 & 53 Minutes to upload each 75GB of Plots. I did find that you needed at least 2 files being uploaded simultaneously to use the bandwidth.

    Rich



  • @RichBC Hmm The lack of cores I can live with but the uplink speed is just too slow.
    I will see if they can change the spec tomorrow.

    edit:
    How many cores show up when you bring up the task manager?
    My processor says common KVM processor 3.6Ghz. Feels like 1Ghz to me though.
    Around 2300 Nonces/min during the plot creation which is around 6X slower than my machine at home (4.5Ghz Skylake)



  • @machasm I did not think there was an option on the Uplink speed, not something I discussed. I thought all the packages were specked at 1Gbps? Feel free to refer to me if it helps. I show 6 cores in Task manager. 5352 nonces / minute

    Rich



  • @RichBC
    I have sent you a chat for PM Rich



  • @RichBC yes, the shutdown resulted from uploading and mining simultaneously. It engaged the acd most of the time. Your idea of simultaneous plotting and mining from one location sounds brilliant! I think the alarm is triggered when reading more than xx% of the time..
    I am uploading now since one week with about 50% active, 50% quiet.



  • @burster
    I had my account locked once so far but I was accessing the drive from multiple locations.
    When you had your account locked were you uploading and mining from the same machine or were you mining from a different machine?
    I am trying to work out if it is the simultaneous upload and download or if it is locked due to accessing from different IPs.


Log in to reply
 

Looks like your connection to Burst - Efficient HDD Mining was lost, please wait while we try to reconnect.