Plotting and Mining on Amazon Cloud Drive



  • Great thread!

    Here's my setup:
    Sony Vaio laptop with i7-4500U @ 1.8GHz
    8 GB RAM
    Win10
    4TB Seagate USB powered HDD

    Software:
    AIO Wallet 0.3.8 rc4 - which is kick-ass. no errors and it uses xplotter 0.7. I think these optimized plots will be key on ACD.

    Netdrive2 - full version. This is used for mining only.

    ACD windows upload software. Using this to upload 25GB plots. Thinking about alternating the size of plots +- 10%.

    I'm currently plotting my 4TB HDD (in 1TB increments), so my ACD work is on hold. My laptop cant handle simultaneous plotting. I'll plot the 25GB files on my C drive about 4 at a time, then upload to ACD during the work day and hopefully come home to uploaded files.

    I'll also reread the posts that Rich mentioned above. I had read them in the past, but always find new tidbits as I gain knowledge.

    Mike



  • i've been working on this for months with no real avail . Ill be watching closely hopefully we can figure it out! i'd love to add an extra 100tb to my portfolio



  • I have just started a new Free EC2 instance to check what mining speed I get. A couple of Blocks have gone by and with my 3.4TB am seeing around 20MB/s read, so the plot goes through in around 50 Seconds.

    There is probably some optimisation of the way the plots are stored in folders that might help a little and it's about 3 times the speed of my own (In the UK) broadband connection. however it's not going to be good enough for a decent sized plot...

    Am not going to leave it running at the moment I still have concerns about accessing Amazon Drive from multiple IP?

    Rich



  • So as Rich stated, the Amazon upload software is bunk.

    Netdrive2 software seems to be working. I've been uploading a 25GB plot for hours now at about 1.2MB/s. Dreadfully slow.

    Is there a way to see the status of the upload in Netdrive2? I have task manager open just to see the network activity. That's the only way I can see if its still working.

    In other news, I downloaded rclone. That's beyond my pay grade. It might be easy for some, but I'm not that skilled. GUI or bust.

    Mike



  • @miraj I also just monitor network activity with NetDrive2. rclone worth persevering with despite being command line. Instructions are quite good and so far has been the most reliable for me.

    Have been experimenting with renting a VPS with a decent internet connection. Definitely much faster uploading, but is likely to work out too expensive because of the spec of system you need to Plot and Optimise before you can do the upload.

    I am in the process of constructing a Batch File that does the Plot, Optimise and then upload to Amazon. Then deletes the files increments the start nonce & runs again. Still testing but I think the complete cycle for 50GB is going to be about 2Hr 40mins.

    However it may all be to no avail as have just had the account locked again. Phone call and it was immediately unlocked, no explanation as to why it was locked, but probably not a coincidence that have been uploading at faster rates for the last 24 Hrs.

    Rich



  • @RichBC

    Rich, I've installed rclone, and I must say, it's pretty sweet. Even a dummy like me has gotten most of the way there. I have a question. I can't copy a file from my c drive to a folder already created on the ACD. I keep getting "409 conflict" error.
    Here's my code:
    rclone copy C:\burst\upload\3051863391608293710_15220161_102336_102336 acd:b1

    I have to use b1, because burst\plots is already used on ACD. Is there a way to copy the file right to acd:burst\plots (acd being the name I used for the amazon cloud drive)? I don't want to have to create a new folder name for every upload.

    Once this works, I'll venture into straight up moving files and start making 50GB plots instead of 25GB....and then get shut down by Amazon.

    Thanks,
    Mike



  • @miraj Not sure on the error but you can copy to any folder on ACD that you want.

    Does folder b1 exist on ACD?

    Other thoughts.

    Run rclone config again and just to check that ACD is there?
    You can just give rclone the path to the directory with the plots, no file name, and it will copy all the plots.

    -v after rclone on the command line gives some verbose output which can be handy
    --Transfers -2 on the command line copies 2 files at once if you bandwidth is not maxed with one file. probably not a problem :-)

    Rich



  • Thanks for your thoughts, Rich.

    I try to copy a plot file to the ACD\burst\plots folder but it says the folder already exists. That is correct, it does already exist on the cloud drive. I just want to put a new file into the folder. rclone doesn't let me so I had to create a new folder called b1. Then when the file is uploaded, I have to move it to burst\plots. Is this the correct code to copy a file to ACD?

    rclone copy C:\burst\upload\3051863391608293710_15220161_102336_102336 acd:burst\plots

    (I called my amazon cloud drive - acd)

    Thanks!
    Mike



  • @miraj Strange all of my copies to ACD have been to existing folders without that message?



  • @RichBC

    Got it. Seems the ACD is very case sensitive. My Burst folder has capital "B". Working great now. Just have to wait 8 hours till its finished!

    Thanks!
    Mike



  • @miraj OK great. Also if you include a directory in the ACD path it will create it.

    Be careful around 50GB there is max file size limit. I have been ok with 48GB & 48.8GB but have also found, at least on slow transfers, that failures grow significantly above 25GB

    Rich



  • @RichBC Is there code in rclone to upload multiple files, but only upload the next file after the previous one is finished? I'm trying to create a bunch of plot files, then when I'm at work, let this run and it uploads all of the files one at a time. I've noticed that my upload connection drops about once every 3 hours or so. It has to keep restarting the uploads. If that happens, i'd only like to lose the file currently being uploaded.

    Thanks,
    MIke



  • @miraj Yes you need to put --transfers 1 on the command line.

    I think the default is 4. From my home connection I use 1, from the VPS I am using 2 or 3 as it then maxes out the Upload bandwidth that is available.

    Rich



  • @RichBC That's exactly what I needed!

    Thanks,
    Mike



  • Quick update on how the Cloud Mining is going.

    To speed up the upload process as an experiment I have rented a VPS for a Month. After some playing I have settled on a minimum spec of 6 Cores, 5GB RAM and 150GB of Hard Disk.

    I have written a Batch file that plots 3 x 25GB plots then optimises them and uploads the 3 plots simultaneously to max out the available bandwidth. This process enables me to upload 1TB every 2 Days. There is about a 10% failure rate on the uploads. I am now up to 9TB.

    So this will work out at about $2.50 / TB which is just about acceptable. I have only been mining the Plots sparingly as I think it might be the Mining process that get's the Drive shut down? I have however been Mining for the last 24 Hours without problem.

    The big issue when a decent amount of plots are up will be the read speed. With a Download Bandwidth of 8MB/s the 9TB takes 4 Minutes to read. :-( I have also been testing Mining from an EC2 Free Instance and get at least twice that speed, but still not really fast enough.

    On the faster speeds I have also seen another problem which is a "error ReadFiles" Sometimes it just a couple of errors but on other occasions you get a complete never ending string of errors. Need to understand this and find a solution otherwise it could be a showstopper. perhaps @blago might have some ideas?

    If I had confidence that Amazon would not shut down the Drive I think this would have the makings of a great low cost, high value Mining Asset as the cost of creating the plots is only $2.50 /TB and the Annual storage costs only an additional $50 for up to 100TB.

    Rich



  • @RichBC said in Plotting and Mining on Amazon Cloud Drive:

    On the faster speeds I have also seen another problem which is a "error ReadFiles" Sometimes it just a couple of errors but on other occasions you get a complete never ending string of errors. Need to understand this and find a solution otherwise it could be a showstopper. perhaps @blago might have some ideas?

    That error is due to the internet connection if i am not wrong... I seen it every once in a while while mining with clouds ;D



  • @gpedro Yes could be and that is what Blago thought. Difficult for me to tell as the problem comes when I am using the Amazon EC2 free Instance, because their Bandwidth is much better than mine.

    Rich



  • @RichBC I think it have something to do with lost packages not really bandwidth tbh ;D



  • Quick up date on progress. I have been concentrating on Plotting rather than Mining. Currently I am up to 17TB of the 25GB files.

    My Batch file now working quite well and I have developed it so that it starts a Group with a 2 Digit prefix to the plot numbers, then plots 80 sequential 25GB files (2TB). It then increments the prefix and does another 2TB of files.

    At the moment it uploads them to a directory called new at ACD and I manually put them into folders of 2TB each. The Batch File could easilly do this but it gives me an opportunity to check things are running ok.

    The batch file also logs to a text file each of the stages along with time stamps to let me monitor progress. Biggest problem with the uploads is the delay in ACD recognising the file is there and that some files have to be uploaded again which slows things considerably.

    However rclone handles this well and the file get there eventually. Heres a typical log file.

    14-Jan-17 Start nonce 941000000 Stagger-102400 Size-6400 
     9:34:01.68 Start Plot 0 
    0:24:40.04
     9:58:41.76 Start Plot 1 
    0:24:41.65
    10:23:23.43 Start Plot 2 
    0:20:31.82
    10:43:55.60 Start Optimise 
    0:44:29.81
    11:28:25.66 Start Upload 
    0:37:37.12
     
    12:06:03.58 Start Plot 3 
    0:20:14.96
    12:26:18.64 Start Plot 4 
    0:20:16.98
    12:46:35.64 Start Plot 5 
    0:19:59.00
    13:06:34.66 Start Optimise 
    0:41:23.09
    13:47:57.86 Start Upload 
    0:41:23.71
     
    14:29:22.45 Start Plot 6 
    0:23:05.12
    14:52:27.80 Start Plot 7 
    0:23:53.23
    15:16:21.28 Start Plot 8 
    0:22:48.35
    15:39:10.41 Start Optimise 
    1:00:40.29
    16:39:50.98 Start Upload 
    1:53:19.70
    

    I do 3 x 25GB plots, log file shows start time for each, then on the following line how long it took. Then the files are optimised, and then uploaded, the 3 files simultaneously as this maxes out the bandwidth at the VPS. So a good sequence of 3 files, 75GB, takes about 2 1/2 Hours, however you will see a longer time for the last upload, indicating there will have been a failure.

    This is from the VPS I am renting, the cost is higher than previously reported due to all the real world problems at both ends of the chain that you hit and I would estimate about $4 / TB but have not redone the Maths.

    Rich



  • @RichBC Hi, I would like to know, how is your experience with your ACD getting shut down by Amazon so far. Is it the plotting from a private VPS to ACD or is it the mining with an EC2 instance from an ACD without interruptions (around the clock)? I got the answer that the reason for my latest shutdown was "continuous operation" ...
    Did you get any problems with them recently, if you did only plotting as you described here?


Log in to reply
 

Looks like your connection to Burst - Efficient HDD Mining was lost, please wait while we try to reconnect.