Hardware for 20x USB 3.0 drives connected to one machine



  • I have had various issues with having 20 USB 3.0 drives hooked up to one machine. 10x 8TB drives, 10x 5TB drives. Sometimes they are stable for weeks, sometime a few drives drop out regularly and I need to reset the drives or the PC, or sometimes several drives are painfully slow (instead of taking 10-15secs to read, they take 7-8minutes). I have tried multiple machines with various results.

    I have 3x 7-port USB hubs. I have also used a couple PCI-E to 4-port USB cards on machines that didn't inherently have USB3 ports but were otherwise pretty juiced up machines. I have tried combinations of plugging some in to the PCIE card, some in to the onboard USB3.0 ports, balancing on the 7-port hubs, etc.

    I am having a hard time determining truly what the bottlenecks are. I know that this is quite demanding on a PC, but I really believe it should be possible.

    Is anyone else doing something similar? Any recommendations for hardware? How about any theory regarding PCIE USB cards, USB3 hubs, vs onboard ports, etc?

    Thanks!



  • Are you using USB 3 cards with 4 controller and 4 ports, or does it have 1 controller divided by 4 ports? I use Sonnet allegro pro with 4 controllers (must be the pro version). When mining, the miner will only read about 0.024 % of your plots, but you will eventually get a bottleneck on your USB 3 controller. I haven't got the exact figures here, but I think you can have about 2-3 HDDs connected to one controller without slowing down read speed, but with 4 or more you will likely get a bottleneck. This may not be true if your HDDs are small capacity, but it will be true with your 5 and 10 TB HDDs. Therefore, your 7-port USB hub will lead to a bottleneck, as it will force all 7 connected HDDs to be read by the same controller. If you're not quite sure what I mean, you should try to learn more about the difference between a USB 3 port and a USB 3 controller.



  • I use 2, 7 port and 1, 4 port usb hubs. I found that especially for the 7 port hubs the power supply was small. I actually got 2 PS from a different 4 port hub. That helped. Also if a drive would drop out I will move it to a different cable or port or both until the system gets stable.



  • @Propagandalf What means bottleneck if I may ask?


  • admin

    @pr0cesor The thing that slows the whole process down. You can pour water out of a jug rapidly, but pouring it out of a bottle it gets slowed down at the bottleneck.


  • admin

    @pr0cesor bottleneck = slowest part, the part limiting others to work faster .. if a bottle had no neck, you could drink beer faster :-)

    @DrTrouble If a pcie card has 4-ports or 7-ports does not matter in speed if all ports are using one controller ... to improve speed ensure you get usb controller cards with dedicated controllers for every port ... no shared. Like this ones: http://geizhals.eu/?cat=ioexpcard&xf=1331_4~621_PCIe+x4

    A system seams to run in trouble on too many usb3 controllers ... @Propagandalf had a lot of issues as he tried to run 5 pcie cards with 4 controllers each.



  • Dang it!!! That is my problem- I was wondering why I'm getting only a combined 100MB/s writing on 3 USB 3.0 drives all plugged into "separate" ports on my USB 3.0 card- duh!! Piece of crap $40 card



  • there are usb 3.0 hubs on ebay. Internal ones up to 7 ports each for PCIE and external ones up to 10 ports each . The internal pc ones are preferable but external ones are ok but best if they are mains powered (for gain, speed and non mains powered external) better speed can be obtained with an internal PCIE SATA card because USB 3.0 is best as a "no more room" option.



  • @ZapbuzZ said in Hardware for 20x USB 3.0 drives connected to one machine:

    there are usb 3.0 hubs on ebay. Internal ones up to 7 ports each for PCIE and external ones up to 10 ports each . The internal pc ones are preferable but external ones are ok but best if they are mains powered (for gain, speed and non mains powered external) better speed can be obtained with an internal PCIE SATA card because USB 3.0 is best as a "no more room" option.

    How many USB 3 controllers you have is a determining factor for your maximum read/write capacity over the USB bus. I believe the internal 7-port card and external 10-port hub you mention only lead back to the same USB 3 controller, and hence the maximum bandwidth is divided over each port.

    The pro version of Sonnet Allegro pro, as a comparison, has 4 unique controllers and 4 ports (one controller for each port), whereas the non-pro version only has 1 controller but 4 ports. If you have many HDDs, having many controllers is the way to go, but it all depends on how good read/write times you need since pro controller cards are expensive.



  • if the miner does intelligent caching then that would improve bandwidth simply working through plots faster



  • @Propagandalf sorry to revive this conversation... however @luxe mentioned

    "A system seams to run in trouble on too many usb3 controllers ... @Propagandalf had a lot of issues as he tried to run 5 pcie cards with 4 controllers each."

    How many maximum were you able to run before you ran into problems? What factors do you think effected that? Can you elaborate more on the issues you found?

    Much appreciate -- looking to splurge on multiple HDs right now and want to make sure I am ready to plug and go.



  • @riceburner55 Hey! No problem.

    I wanted to control 106 HDDs through one rig, but I ended up splitting the load over three rigs, it will save you a world of trouble! Now I only use two multi controller cards per rig, but I could probably use three.

    I think sharing my chat with Sonnet Tech might help (producer of multi controller cards). It's the most detailed explanation I can think of:

    Dear Sonnet

    I have bought 5x Sonnet Allegro pro USB 3 controller cards. For some reason, I am not able to boot my computer (not even to BIOS) if I have 3 or more cards connected to the PCIe slots. 2 cards connected work fine, and I have also tried swapping the cards and slots, so there seems to be nothing wrong with them.

    Here are my system specs that I think are relevant:

    Mobo: x99-E ws/USB 3.1 (https://www.asus.com/us/Motherboards/X99E_WSUSB_31/specifications/)
    CPU: CPU Xeon E5-1620 v4 3.5 GHz (https://ark.intel.com/products/92991/Intel-Xeon-Processor-E5-1620-v4-10M-Cache-3_50-GHz)
    GPU: MSI Radeon rx470 8GB (https://www.msi.com/Graphics-card/Radeon-RX-470-GAMING-X-8G.html#hero-overview)

    Do you have any idea what might be the problem? Is my issue 'normal' when using more than 2 cards?

    Reply:

    In answer to your question, this isn’t a “normal” issue; however, we have seen issues like this with PCs in the past.

    In design, the Sonnet Allegro Pro USB 3.0 card has 4 independent USB 3.0 controllers on each board. When installed the Computer sees the card actually as 4 separate PCIe cards. In effect, when 3 cards are installed into a single computer, that computer thinks that it has 12 USB 3.0 PCIe cards installed.

    In some instances that configuration will still work because the particular motherboard will have enough resources available for all the cards alongside the resources required for its own built-in devices.

    That’s not always the case though, and because there are so many different manufactures of PC equipment, there’s no way to be certain outside of trying a particular setup. With your current configuration, there aren't enough system resources to operate the additional card and the conflict causes the hang at boot.

    Unfortunately, there usually isn’t a way to resolve this conflict. Sometimes I have been able to disable unused on-board devices to free up enough resources to solve the issue, but since there’s no way to uncover where the conflict is it may be impossible to resolve.

    So I hope this helps answer your questions, and thanks for your support of Sonnet!

    Follow up:

    Thanks for the reply.

    I managed to get three cards working by disabling everything I could of on-board devices and similar, but I noticed that the one card only detected devices through 2 out of 4 ports. Do you reckon that this is also due to a lack of resources for 2 of 4 controllers (on one card), based on how you say these cards are 'treated' by the system? It is interesting that I got card #3 working after disabling the serial port only, and after that I disabled lots of other things and on-board USB 3 controllers even, but strangely enough I was not able to use a fourth card.

    I tried the controller cards on a cheaper, inferior motherboard (ASRock 970 Extreme 4), and I was able to use four controller cards after disabling on-board devices. I just don't get why the 700 USD motherboard is the one that cannot handle four or five cards and not the other way around. Are you sure it is not possible to somehow reset the 'master list' of IRQ / PCI bus allocation of resources? I have a hunch that I have already freed up resources, but that they are not being used at all or not in the right place.

    If you know of a particular motherboard that can handle four or more of these controller cards, please let me know.

    A final question: Do these cards require additional power when I am using several at the same time? I have powered PCIe risers, and also a PSU 6-pin EATX plug that I could attach to the motherboard. I have already tried this without success, but I am just curious whether extra power is needed or not.

    Reply:

    Unfortunately the huge variety of PC motherboards makes it difficult to determine which ones will work and which won’t. Really the only way to try them out.

    On the other issue, you are correct: It’s a strange combination, but more expensive motherboards are often more difficult to to work with because their on-board devices utilize more system resources. The less powerful motherboards have fewer features built-in and so surprisingly they have more resources available for additional cards. It’s a balancing act, and it’s not uncommon for devices to have partial operation (like the 2 of 4 ports) because that was the absolute limit of resources that could be freed up.

    Also, allocation of the system resources is done at startup automatically by the computer’s BIOS, so there’s no way for users to change that in the normal course of things.

    On the power issue, it’s very important to insure your motherboard has the recommended specification for its power supply. The Allegro card is built to draw 25 Watts from the x4 PCIe slot (that is the Intel specification for an x4 PCIe slot). This allows it to supply the required 4.5 Watts to each USB 3.0 port, as well as enough power for its own operation. Outside of that, no additional power is required, but again, every card must be provided the 25 Watt specification.

    Follow up:

    Once again, thanks for the reply. I just have one final follow-up question. What if I had a server motherboard which is able to hold two CPUs, would that not give me more resources for tackling this issue? Or are we talking about a different set of resources than what we are provided by the CPUs?

    Reply:

    No, that wouldn’t affect the cards.

    This is an issue of PCIe cards and on-board devices all utilizing the same space, and attempting to find the necessary resources alongside each other. Doubling the processors won't increase the available space.



  • @Propagandalf This is perfect. Thank you so much. I will stick with 2-3 cards per device to not cause any issues.

    106 HDDs per one rig seems a little aggressive LOL.... sounds like one hell of a set up you got going on over there.

    Happy mining :)



  • Yes I thank you too


Log in to reply
 

Looks like your connection to Burst - Efficient HDD Mining was lost, please wait while we try to reconnect.