Please feel free to send documentation patches to the vbox-dev mailing list.
Mechanics of host-guest dynamic resizing
In normal use, with the standard graphical user interface and with Guest Additions installed, VirtualBox resizes guest desktops to match whenever the host window is resized. This document describes the way the mechanism is supposed to - and usually does - work. It also points out the tricky parts which might potentially fail.
The guest video device
VirtualBox virtual machines contain an emulated graphics card which can currently support up to 64 monitor outputs (the actual number is set by the user before starting the machine). Each output can be set to any resolution which the configured video RAM size will support (potentially using overlapping video RAM areas). Normally it will be controlled by the Guest Additions video driver, but it can also be controlled by a VGA driver or using the VESA BIOS, which can only set a limit set of the resolutions that the card can handle, and only on the first screen. When the guest sets a new resolution on a given monitor output the user interface resizes the matching window to fit. This only happens once for each resolution change. If it appears to be repeating, the guest has probably got stuck setting and resetting resolutions.
Resize requests from the host
The VirtualBox graphics card has a mechanism to transmit resize requests from the host to the guest operating system. Generally the Guest Additions graphics driver tells the guest that a monitor of the given size has been plugged into the matching monitor output. If automatic resizing is enabled, then when the user directly or indirectly resizes a virtual machine screen window on the host system (an example of indirect resizing would be a window manager adjusting the size) the graphical interface sends one of these "size hints" to the guest. The expected result is that the guest operating system will almost immediately resize that output to the size requested. The user interface will see that the new output size matches the window's size and will not attempt to change it.
In theory the guest could decide to set a slightly different size, perhaps just by one or two pixels, and the host could try to insist on the original one, but in practice this has not been observed. What is more common is that the guest ignores the request for some reason - for example if the user has accidentally told GNOME Shell in a guest that they only want a certain resolution. Another known problem is that on older Linux guests without kernel graphics drivers we send an ACPI monitor change notification, which a few older guests interpret as a request to cycle to the next available resolution after the resize has already taken place. No modern Linux system is known to be affected, and as of X.Org Server 1.19 in the guest we require use of the kernel driver.