PCIe lanes, usb 3 controller cards, IRQ - troubleshooting
Does anyone know if creating a virtual machine (VM) can solve this problem somehow? I'm not too familiar with VMs, but if they function like a physical system even though they are virtual, then maybe I could let a VM "take over" some of the PCIe USB 3 controller cards, so that they will not conflict with POST and boot on my physical machine? I'm thinking out loud here, hoping someone else knows more about it than I do.
Will not work. Your IRQ problem is on a hardware level and not software.
@Propagandalf if you have an os installed on the pc, setup VNC (remote control) on it. Then you can disable the mouse and keyboard in the bios giving you 2 extra IRQ's. You will need another pc or laptop to connect to it to control it.
@Propagandalf Shot in the dark, but is it possible that just ONE of the cards is bad? Have you tried swapping out the third card for one of the others?
I also notice there's no SATA or Molex power connector on that Allegro Pro USB3.0 PCIe card. Seems unlikely, but I wonder if the PCI bus itself can't allocate enough power to all the cards.
Not even by using VT-d? https://software.intel.com/en-us/blogs/2009/02/24/step-by-step-guide-on-how-to-enable-vt-d-and-perform-direct-device-assignment. You're probably right, since I can't pass POST, but I was thinking that if I was able to first boot using 3 or less cards, then use something like VT-d, that it would free up or change IRQ channels somehow.
I found out how to disable ACPI and revert to 'standard pc', but apparently I will likely run into a lot of BSODs if I mess around like that and manually change channels and ranges.
I don't have PS/2 ports, if that was what you meant? The closest I can get to disable my USB mouse and keyboard would be to deactive USB legacy support I think? If I understand it correctly, it disables USB detection during startup, but then they will be detected once my OS has booted (Windows 10).
Maybe, but I don't think so as I have swapped them around earlier. But, I will try again tomorrow in case you are right! Apparently, similar cards who have SATA or Molex use that extra power to supply connected USB devices that do not have their own adapters, and not to power the card itself as it draws from PCI bus. Those cards will work even if you do not connect SATA or Molex, so I don't think that is an issue with the Sonnet Allegros when they don't have additional connectors. Still, it is a possibility.
I have contacted both Sonnet Tech and Fresco Logic (chipset manufacturer) for additional support. I realize that encountering computer problems makes you learn a lot of new stuff, but it feels like there's always something that goes wrong, and always with very bad timing.
Surely there must be a way to force the OS to reset the IRQ distribution 'master list' other than clearing CMOS and reinstalling the OS, but I still haven't found it. When I go into device manager, it shows that two of the cards are installed in PCIe slot 6 and one card in PCIe slot 4. This may be normal and may mean nothing, but I suspect this means that the 'master list' is not set up in an optimal way. The OS is supposed to handle this like a boss, but since people have IRQ issues nonetheless, finding a way to reset the distribution could be a viable solution, or not.
It may be just temporary, but one of the cards only detected connected devices through 2 out of 4 ports. I will be sure to investigate this further.
After reading everything maybe its the motherboard with a bad pcie slot. You say two cards work. Have you tried the 2 cards in every slot to rule a bad one out. start with the third and 4th slot.
I contacted Sonnet customer support (manufacturer of the controller cards), explained the situation, and got this reply:
"In answer to your question, this isn’t a “normal” issue; however, we have seen issues like this with PCs in the past.
In design, the Sonnet Allegro Pro USB 3.0 card has 4 independent USB 3.0 controllers on each board. When installed the Computer sees the card actually as 4 separate PCIe cards. In effect, when 3 cards are installed into a single computer, that computer thinks that it has 12 USB 3.0 PCIe cards installed.
In some instances that configuration will still work because the particular motherboard will have enough resources available for all the cards alongside the resources required for its own built-in devices.
That’s not always the case though, and because there are so many different manufactures of PC equipment, there’s no way to be certain outside of trying a particular setup. With your current configuration, there aren't enough system resources to operate the additional card and the conflict causes the hang at boot.
Unfortunately, there usually isn’t a way to resolve this conflict. Sometimes I have been able to disable unused on-board devices to free up enough resources to solve the issue, but since there’s no way to uncover where the conflict is it may be impossible to resolve."
I followed up and asked how come an expensive board could only handle 3 controller cards (one card only got 2 out of 4 ports working), and an inexpensive board was able to handle 4 controller cards. I also asked about power supply and power draw. I got this reply:
"Unfortunately the huge variety of PC motherboards makes it difficult to determine which ones will work and which won’t. Really the only way to try them out.
On the other issue, you are correct: It’s a strange combination, but more expensive motherboards are often more difficult to to work with because their on-board devices utilize more system resources. The less powerful motherboards have fewer features built-in and so surprisingly they have more resources available for additional cards. It’s a balancing act, and it’s not uncommon for devices to have partial operation (like the 2 of 4 ports) because that was the absolute limit of resources that could be freed up.
Also, allocation of the system resources is done at startup automatically by the computer’s BIOS, so there’s no way for users to change that in the normal course of things.
On the power issue, it’s very important to insure your motherboard has the recommended specification for its power supply. The Allegro card is built to draw 25 Watts from the x4 PCIe slot (that is the Intel specification for an x4 PCIe slot). This allows it to supply the required 4.5 Watts to each USB 3.0 port, as well as enough power for its own operation. Outside of that, no additional power is required, but again, every card must be provided the 25 Watt specification."
So, I guess this sums up everything, and I have to give up getting 5 controller cards to work in one system. For what it's worth, I have sores and scratches all over my hands from almost taking apart both my systems, swapping around cards, PSUs, cables and what not. I'm a little bit disappointed by the conclusion, but I have also learned a lot, so it's not all bad. I want to thank you all for helping me along the way.
The big question remains, though: What kind of motherboard will be 'simple' enough to be able to handle as many controller cards as possible?
@Propagandalf You might have better luck with a server grade motherboard with dual CPU sockets. I figure two CPUs should have twice as many resources, right?
This motherboard looks like it would support all those extra cards, but it's kind of old being on the 2011 socket, uses DDR3, only USB2.0 native, etc.
ASRock EP2C602 SSI EEB Server Motherboard Dual LGA 2011 Intel C602
I still think building two separate machines with current gen hardware would be the better way to go, though. Less scalability issues and more flexibility for future upgrades or repurposing. Also, I figure if my machine can scan 50TB of plots in about 20 seconds, then a second identical machine could do the same with another 50TB in 20 seconds, whereas one machine scanning 100TB might take closer to 30 seconds total (just an example).
I figure two CPUs should have twice as many resources, right?
I want to think that is true, but then maybe there are other 'on-board' restrictions that would limit the resource distribution regardless of the CPU power... It would be interesting to test the theory using one server motherboard, first with just one CPU, and then install another and see if it could handle more cards. But, I don't have the resources to carry out such a test for the time being, so hopefully someone else can test and give an update at some point. =)
I really don't think it can be done. As you heard from the manufacturer, each card is like 4 controllers so 4x the resources needed.
Your motherboard is a damn good one and expensive ~$700 so IMO the mobo is not the problem.
Work with 3 cards, I guarantee with that you would be able to read 50-60TB per port easily for a total of 600-700TB
Definitely, and if I ever want even better read speeds with more cards, I can simply spread them out over more computers.