Booting iPXE using Intel vNIC fails - regression since 5.0.26 => Fixed in SVN

[Robin Smidsrød]

I recently tried to upgrade from VirtualBox 5.0.26 to 5.1.4, and I had to rollback to 5.0.26 again, because I couldn't boot over the network using either my custom-built 56KB iPXE ROM or a stock version of ipxe.dsk (floppy image) booted via floppy emulation. In both cases when I start iPXE shell and run ifstat all of the packets counters show 0. If I run "dhcp" and run ifstat again it fails with error ​http://ipxe.org/err/040ee1, which indicates that no DHCP response or IPv6 SLAAC response reached the VM. What's actually interesting is that my DHCP server sees the DHCP DISCOVER and returns the DHCP OFFER, but the VM never sees this DHCP OFFER. I tried to enable --nictrace on the VM, which confirmed that the DHCP OFFER was present at that point, but iPXE never received it. Rebuilding ipxe with DEBUG=intel:3 didn't give any obvious pointers to what was wrong.

If I booted with virtio-net instead, everything worked as expected. My assumption is that the iPXE intel driver is no longer capable to speak with the new emulated intel vnic in 5.1.4. Just to confirm that there are no network issues here I booted up a Ubuntu VM and it had no issues communicating with the network and get a DHCP lease. I also tried all three intel vnics, and they behaved slightly different, but all broken in some way which prevented network packets from reaching iPXE. So my assumption is that the problem might be somewhere in the receive (RX) path.

Attached are log files for both 5.0.26 and 5.1.4. Hopefully you'll be able to figure out what the problem is.

[Robin Smidsrød]

VirtualBox 5.0.26 log file

[Robin Smidsrød]

VirtualBox 5.1.4 log file

[Frank Mehnert]

Reproduced. We found the resonsible change (r61691). We need to find out what's wrong with this change.

[Robin Smidsrød]

I talked with Michal Brown (mcb30, lead developer of iPXE) and he had this to say about the issue:

• robinsmidsrod mcb30: I've got a bug on vbox 5.1.4 wrt. intel vnic
• robinsmidsrod one of the comments points to this commit in vbox which reworks e1000 irq handling. Can you see what change in this commit broke the intel driver?
• robinsmidsrod should they fix their code or should the ipxe intel driver be more robust?
• robinsmidsrod maybe both?
• mcb30 robinsmidsrod: from a quick look, it seems as though the virtual NIC defaults to having interrupt throttling enabled
• mcb30 which I don't think can be correct
• mcb30 Also, it looks as though interrupt throttling is controlled by config file directives, rather than whatever the driver programs in to the registers...
• mcb30 I'm digging through the code. I think the config file flags function as global disables, allowing the user to disable throttling even when the driver requests it
• mcb30 which is fine
• mcb30 The value of the ITR register should still be zero when iPXE is running
• mcb30 which means that e1kRaiseInterrupt() should always take the "we are delivering the interrupt now" path, regardless of the config file flags
• mcb30 but this is irrelevant for your use case anyway, since you're not using undionly.kpxe (and hence not relying on interrupts)
• mcb30 is thinking aloud
• mcb30 robinsmidsrod: found it!
• mcb30 In e1kRegReadICR(), that commit added this code:
• mcb30 /* Do not return masked bits. */
• mcb30 value &= IMS;
• robinsmidsrod I was also looking at that thinking "is that correct?"
• mcb30 which means that reading from ICR will never report interrupts that are currently masked off
• mcb30 It's not correct
• mcb30 and from the comment just below that, it looks as though this change should also break QNX
• robinsmidsrod got a suggestion for a fix?
• mcb30 My guess is that just deleting those two lines should fix iPXE
• mcb30 but I don't yet understand why they added those lines
• mcb30 so there's a reasonable chance that taking them out will break something else in vbox
• robinsmidsrod mcb30: well, at least it is a step in a right direction - hopefully some vbox developers might answer that question
• mcb30 robinsmidsrod: am just tracking down the part of the NIC spec...
• mcb30 robinsmidsrod: from the 8254x Software Developer's Manual (​http://www.intel.com/content/dam/doc/manual/pci-pci-x-family-gbe-controllers-software-dev-manual.pdf):
• mcb30 section 2.6 (page 30):
• mcb30 Interrupts appear on PCI only if the interrupt cause bit is a 1b, and the corresponding interrupt mask bit is a 1b. Software can block assertion of the interrupt wire by clearing the bit in the mask register. The cause bit stores the interrupt event regardless of the state of the mask bit.
• mcb30 so reading from ICR should not be masked by IMS
• mcb30 iPXE relies on this behaviour, and from the comments in DevE1000.cpp, it sounds as though QNX also relies on it
• robinsmidsrod okay, so they're breaking intel spec behavior with that change, so that line should be possible to just remove, right?
• robinsmidsrod or must something be put in place?
• mcb30 We need to know why the author found it necessary to add those two lines in the first place. It's possible they were used to work around some other bug elsewhere.
• mcb30 But yes, they are breaking the Intel spec behaviour with that change
• mcb30 and those lines should definitely be removed

[Aleksey Ilyushin]

I guess I missed that section of the manual this time. I was considering a potential case of shared interrupt line, when a device other than e1000 actually raises an interrupt while e1000 driver (Linux) is still in NAPI polling mode. It will re-enable interrupts prematurely in such case. Sorry about the trouble. By trying to solve a potential issue, I caused a real one. By the way, QNX guest worked for me somehow, as I tested with one. Which was surprising a bit, as I remembered that I had had to revise interrupt logic during original development because QNX had used to hang in ICR polling loop with interrupts masked.

[Aleksey Ilyushin]

I confirm that removing value &= IMS; fixes the issue with iPXE. This will be included into the next maintenance release. Thanks a lot for actually solving the problem.

[Robin Smidsrød]

Great to hear that it is fixed and will be in the next release! All of us in the iPXE community are very happy for a quick fix to the issue. We heard about the problem a little while back, but weren't in a position to actually file a proper bug report until now.

I guess the only question left to ask is when the next release is expected to hit the streets?

Again, thanks a bunch! It's always nice to see different open-source communities coming together to solve shared issues.

[Frank Mehnert]

The most recent 5.1.x test builds contain the fix. Sorry, I cannot provide any ETA for the next 5.1.x maintenance release.

[Frank Mehnert]

Fix is part of VBox 5.1.6.