Failure to create parition based VMDK file

[Paul de Vrieze]

When I try to create a vmdk file for my windows partition (on linux) I get the following output:

 $ vboxmanage internalcommands createrawvmdk -filename /home/me/.VirtualBox/VDI/sda2 \
-rawdisk /dev/sda -partitions 2 -mbr /boot/backup_mbr -register
VirtualBox Command Line Management Interface Version 1.5.0
(C) 2005-2007 innotek GmbH
All rights reserved.

Overlapping partition description areas. Aborting
Error reading the partition information from '/dev/sda'

Creating a full-disk image does work. (without the mbr stuff)

[Frank Mehnert]

There were fixes in the latest versions so please try version 1.6.2 and reopen this bug if the problem is still present.

[Shaji]

I am still seeing the same issue with 1.6.2

[Terry Ellison]

If you look at the source of VBoxInternalManage.cpp, there is a logic flaw in lines 595-641. These do a simple exchange sort on the pPart.aPartitions on the column uStart; at the same time doing some order checks. The logic here is wrong, and fails if there is an extended partition, except by a double bug where it works if the extended partition is the last partition.

There is no requirement for a Logical partition to be the last partition on a disk, so this is definitely wrong.

This code is common to listpartitions so also occurs there. I can propse the code changes if you want. (I have already signed the Sun JCA).

[Terry Ellison]

See thread ​http://forums.virtualbox.org/viewtopic.php?p=26912#26912 for further discussion.

[Klaus Espenlaub]

Unfortunately the further discussion in the forum contains not much more useful information. Also all the cases have the extended partition as the last primary partition, so your claim that there's another bug which makes this case valid doesn't seem to be entirely correct. In fact I have a system here where the extended partition is number 2, before that is a primary NTFS and after that is a primary Linux partition. So either your analysis is wrong or there must be another bug.

I've tried recreating the exact same partition layout as in the failed cases but couldn't really get there. Without a way to reproduce this I can't quickly debug this issue. What would help me a lot is if I could reproduce it myself. However it's seriously complicated to extract the partitioning information of a disk with an extended partition so that I can recreate the exact same layout. A first step would be providing the more detailed partitioning information provided by fdisk if you go to expert mode (using command 'e'), and printing the partition table in this mode (using command 'p'). Getting out of fdisk works as usual in this mode (command 'q').

So let's work together and squash this bug.

[Terry Ellison]

Klaus, if my assertion is correct then we should be able to excite this logic flaw repeatedly using test vectors into a small unit test stub based on lines 595-641. Let me have a play and I'll come back with an example or a red face :-)

[Shaji]

Klaus, I think you will get the hint here:

Since I do not have 4th primary partition, the line 4 is returning all zeros. Probably this needs to be handled correctly by virtualbox.

Disk /dev/sda: 255 heads, 63 sectors, 19457 cylinders

Nr AF Hd Sec Cyl Hd Sec Cyl Start Size ID

1 80 1 1 0 254 63 1023 63 61448562 07 2 00 254 63 1023 254 63 1023 61448625 88293240 83 3 00 254 63 1023 254 63 1023 149741865 162834840 0f 4 00 0 0 0 0 0 0 0 0 00 5 00 254 63 1023 254 63 1023 126 3871539 82 6 00 1 1 1023 254 63 1023 3871727 71681967 07 7 00 254 63 1023 254 63 1023 63 31246362 83 8 00 254 63 1023 254 63 1023 63 56034657 83

[Shaji]

Disk /dev/sda: 255 heads, 63 sectors, 19457 cylinders

Nr AF  Hd Sec  Cyl  Hd Sec  Cyl     Start      Size ID
 1 80   1   1    0 254  63 1023         63   61448562 07 
 2 00 254  63 1023 254  63 1023   61448625   88293240 83
 3 00 254  63 1023 254  63 1023  149741865  162834840 0f
 4 00   0   0    0   0   0    0          0          0 00 
 5 00 254  63 1023 254  63 1023        126    3871539 82 
 6 00   1   1 1023 254  63 1023    3871727   71681967 07 
 7 00 254  63 1023 254  63 1023         63   31246362 83 
 8 00 254  63 1023 254  63 1023         63   56034657 83 

[Terry Ellison]

OK, I understand this code. It is a LOT clearer than the source code for sfdisk, but still rather more complicated than necessary. (I used sfdisk as a reference, together with the Wiki articles for MBR and EBR.)

Lines 594-641 do an exchange sort of this table on the start field, but with validation code included in lines 625-641. This code would have been a lot easier to understand if it had been coded in two sections: the nested for loop just doing a bare exchange sort, and a separate validation pass to detect overlaps etc.

However at line 625, this at a point in the outer loop of the exchange sort. At pass i, entries (0..i) are now in ascending order of start(i) with the remaining entries in original order. prevEnd has been set in a previous loop to Start(k) + Size(k) where k = was the last primary or logical partition in the entries (0..i-1).

The treatment of the fields uPartDataStart and cPartDataSectors is not consistent and they do not have a clear definition. They are give calculated values for logical partitions, but default to zero otherwise. Hence the line 625 conditional is synonymous to "if extended partition". The test at 627 is meaningful, but there is a logic flaw introduced in 632+634 since:

  .uStart  = uStart + uOffset + uStartOffset                (From line 570)
  uPrevEnd = .uPartDataStart + .pcPartDataSectors           (Line 623)
           = uStart + uOffset + min(uStartOffset, 63);      (Line 574-575)
		   = .uStart                       if uStartOffset <= 63
		   = .uStart + (uStartOffset - 63) if uStartOffset > 63

Hence the test at line 634 will fire iff uStartOffset > 63 which is not what was intended. The single EBR sector at the start of a logical partition will cause alignment problems for the following data area, so this offset allows the data area to be track or cylinder aligned. The convention seems to be that this field is T-1 sectors where T is the number of sectors per track, to track align the data area. In the case of the Linux partition utilities, this offset is 63 on large devices. However, this is a convention and not a specification requirement. Indeed on modern high performance file systems, there is a rationale to make this greater than 63 if you wanted to keep true track alignment for logical partitions.

The reasoning behind the MIN tests at lines 574 and 645 must be challenged. Why? I am also at a lost as to the intention of line 632.

[Terry Ellison]

nvshaji, Some of the EBR discussion is about Logical Partition allocation schemes which result in nested LPs. Putting together your two fsdisk dumps, that's what you have on your HDD. L2 is nested inside L1. The sector allocations are as follows:

ontent  #sectors      Offset    Start    End   Exp Start   Exp Size
MBR             1           0
pad            62           1
P1       61448562          63       1    3825          63   61448562
P2       88293240    61448625    3826    9321    61448625   88293240
P3      162834840   149741865    9322   19457   149741865          0
EBR1            1   149741865                         126    3871539
EBR2            1   149741866                     3871727   71681967
pad           124   149741867
L1        3871539   149741991    9322    9562
pad            63   153613530
L2       71681967   153613593    9563   14024
EBR3            1   225295560                          63   31246363
pad            62   225295561
L3       31246362   225295623   14025   15969
EBR4            1   256541985                          63   56034657
pad            62   256541986
L4       56034657   256542048   15970   19457
                    312576705

Even though the Linux Kernel tolerates this nested EBR1,EBR2,L1,L2 sequence, this algo throws the error you get.

[Klaus Espenlaub]

That weirdly nested partition case is definitely not accepted - this is intentional as I thought this partitioning violates the spec. I don't care much about the "Linux accepts it" argument, what I'm worried about is how to handle this case properly in the raw partition access code. That's not yet completely clear to me - and thus the code refuses it.

The problem this code is trying to solve is distilling out which areas of the disk should be blocked, which areas should be passed through, and which should be stored in the small image file holding the partitioning information. The actual VMDK backend code is pretty simple minded and expects everything pre-chewed in the right order in RawDescriptor. And the consistency checking should spot anything the backend could possibly choke on. The data structure declaration for VBOXHDDRAWPART contains the description of what the fields mean. The situation which seems to be the problem with those oddly nested logical partitions is that it's not obvious how to represent them with this information without breaking the ordering assumptions of the VMDK backend code. So maybe that also needs to be fixed.

And BTW, the unused entry shouldn't play any role here as it's filtered out before getting to those checks.

I've cleaned up the code a bit and removed the RT_MIN(..., 63) as I couldn't figure out the reason for them either. This doesn't yet solve the problem for the weirdly ordered EBR/partitions, but maybe is a start.

[Terry Ellison]

Klaus, you say:

"I thought this partitioning violates the spec"

and this goes to the nub of the issue here: there isn't a definitive ISO standard or RFC that I can find. There only seem to be the implicit definitions of what the standard Unix/Linux/Microsoft utilities allow you to do and what the respective kernels/drivers tolerate. About the best reference that I've come across is ​http://tldp.org/HOWTO/Partition/index.html. I suppose there is also the issue of VMware interoperability.

I also sort of disagree with your statements about ordering. OK, EBR1, EBR2, L1, L2, EBR3, L3 is a bit pathelogical and won't occur if you are using the NTFS Disk Management tools, but it can occur using some of the Linux tools. Also you can easily get EBR2, L2, EBR1, L1, EBR3, L3 as a result of resizing / deleteing / inserting partitions. Your current algos do and should tolerate this, this second case. The first case should at least create a meaningful error message with is explain in the UG or wiki. In this case, the solution is to use sfdisk advanced mode to fix the partition tables. I think that the integrity rules are:

• There can be up to 4 Physical + Extended Partitions

• There can only be one Extended Partition

• These partitions must be disjoint and lie within the physical limits of the HDD

• The Extended partition can contain multiple Logical Partitions, with each Logical Partition proceeded by its EBR

• The Logical Partitions and EBRs must be disjoint and lie within the limits of the Extended Partition

I think that you wish to include

• No second Logical Partition and EBR can fall between a Logical Partition and its EBR

[g2g591]

This bug seems to remain unfixed as of virtualbox 2.1.0, pastebin of my partitioning and command line used to attempt to create raw vmdk @ ​http://rafb.net/p/kT3OUy11.html

[Nils Fredrik Gjerull]

I was hit by this bug when I tried to make a raw disk vmdk file. I have an extended partition between two primary partitions, for historical reasons :).

Partition table

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        2196    17639338+   7  HPFS/NTFS
/dev/sda2            2197        7287    40893457+   5  Extended
/dev/sda3            7288        7296       72292+  83  Linux
/dev/sda5            2197        2325     1036129+  82  Linux swap / Solaris
/dev/sda6            2327        5584    26169853+  83  Linux
/dev/sda7            5586        7287    13671315   83  Linux

Advanced partition table

Disk /dev/sda: 255 heads, 63 sectors, 7296 cylinders

Nr AF  Hd Sec  Cyl  Hd Sec  Cyl     Start      Size ID
 1 80   1   1    0 254  63 1023         63   35278677 07
 2 00 254  63 1023 254  63 1023   35278740   81786915 05
 3 00 254  63 1023 254  63 1023  117065655     144585 83
 4 00   0   0    0   0   0    0          0          0 00
 5 00 254  63 1023 254  63 1023        126    2072259 82
 6 00 254  63 1023 254  63 1023    2088512   52339707 83
 7 00 254  63 1023 254  63 1023   52355835   27342630 83

[Ivan]

I have this issue

My partition table looks fine:

Disk /dev/sda: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x95aa95aa

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        9526    76517563+   7  HPFS/NTFS
/dev/sda2            9527       10017     3943957+  83  Linux
/dev/sda3           10018       14593    36756720    f  W95 Ext'd (LBA)
/dev/sda5           10018       10080      505984+  82  Linux swap / Solaris
/dev/sda6           10081       11314     9912073+  83  Linux
/dev/sda7           11315       14593    26338536    7  HPFS/NTFS

and the output even with root and even with user is:

LucaBrasi:~# VBoxManage internalcommands createrawvmdk -filename /home/ivan/.VirtualBox/Windows.vmdk -rawdisk /dev/sda -partitions 1,7 -mbr /home/ivan/.VirtualBox/Windows.mbr -register -relative
VirtualBox Command Line Management Interface Version 2.2.0
(C) 2005-2009 Sun Microsystems, Inc.
All rights reserved.

Overlapping partition description areas. Aborting
Error reading the partition information from '/dev/sda'
The raw disk vmdk file was not created

Prior I was affeted by

Error while creating the raw disk VMDK: VERR_INVALID_PARAMETER

that I solved with fdisk.

[saerdnaer]

I hit the same bug today. It showed the same symtoms as in kintaro77 case. (no partion 4, sda3 = extenended partition of type 0x0f W95 Ext'd (LBA). I had some free space (~1MB) before the extended partition, so i created an partition sda4 from that and changed the type of the extened one form 0x0f to 0x05 Extended.

After this procedure it worked as intented.

But this is only some kind of workaround, so please fix this bug until >2 years of is existence.

[Markus Schaber]

I have an identical problem. The partition I wantet to virtualize is /dev/sda1 - the partition table looks like:

Disk /dev/sda: 320.1 GB, 320072933376 bytes 255 heads, 63 sectors/track, 38913 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0xb0c9b0c9

Device Boot Start End Blocks Id System

/dev/sda1 * 2 12749 102398310 7 HPFS/NTFS /dev/sda2 38783 38913 1052257+ 83 Linux /dev/sda3 12750 25497 102398310 83 Linux /dev/sda4 25498 38782 106711762+ 5 Extended /dev/sda5 26030 38782 102438472+ 83 Linux /dev/sda6 25498 26028 4265194+ 83 Linux

Partition table entries are not in disk order

[Cefn Hoile]

I experience the same error message on 3.1.6 r59338 - the latest available stable download. Is there a workaround to simply author the vmdk, given that the partitions are not overlapping?

I'm following the instructions as suggested here... ​http://ubuntuforums.org/showthread.php?t=769883 ...and I'm trying to create a VMDK for my /dev/sda1 partition.

The error output I get is as follows...

cefn@cefn-ubuntu-lenovo:~$ VBoxManage internalcommands createrawvmdk -filename ~/.VirtualBox/WinHD.vmdk -rawdisk /dev/sda -partitions 1 -relative -register
Sun VirtualBox Command Line Management Interface Version 3.1.6
(C) 2005-2010 Sun Microsystems, Inc.
All rights reserved.

Overlapping partition description areas. Aborting
Error reading the partition information from '/dev/sda'
The raw disk vmdk file was not created

My fdisk output looks like this...

Disk /dev/sda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xba1c09cd

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        3060    24579418+   7  HPFS/NTFS
/dev/sda2           18170       19457    10342080    7  HPFS/NTFS
Partition 2 does not end on cylinder boundary.
/dev/sda3            3061        6120    24579450    7  HPFS/NTFS
/dev/sda4            6121       18169    96783592+   5  Extended
/dev/sda5            6607       18169    92879797+  83  Linux
/dev/sda6            6121        6605     3895699+  82  Linux swap / Solaris

Partition table entries are not in disk order

[Klaus Espenlaub]

A raw partition VMDK is pretty difficult to create manually. The reason is that the sectors containing partitioning information are copied to a rather special flat image, and the actual partition contents are directly referenced.

The better approach would be to debug the code which analyzes the partition table and falsely triggers the error message. There are certainly no overlapping areas, but somehow the totally messed up partition ordering confuses the code.

[Cefn Hoile]

Update: I now have a working installation, but only after completely repartitioning and reinstalling my main OS in a way which seems pretty heavy duty just to be able to address another partition in a VMDK.

Finally I ended up having to assign all 4 partitions as primary partitions, including a 4Gig swap drive, as primaries, to get around the original bug. Had to rewrite my MBR about 3 times to experiment with what was actually causing boot problems.

Turned out I had encountered a separate Virtualbox bug in parallel, related somehow to my Thinkpad disk geometry, that needed the number of heads to be modified to 240 from 255 (the introspected value) in order to avoid a 'disk read error' when booting Windows XP.

Could I suggest that the priority be increased, given that this bug means it is impossible to reliably run a whole class of installed OSs, and, though separate from the final major bug I encountered, added enough uncertainty about the stability of my partition-based setup to make me nearly give up altogether.

[Bruno]

Same problem here on a Dell Vostro 1520. I have gparted working perfectly, but VBoxManage is reporting overlapping partitions.

Is there any workaround for the problem? 3.1.8 solved that?

[Klaus Espenlaub]

No, the 3.1.8 changelog says nothing about fixes in this area, so it was unchanged.

Keep your fingers crossed for 3.2.0.

[Klaus Espenlaub]

Good news, the change made it into 3.2.0 just before the deadline. Can you test whether the new code behaves correctly now? By design it should be able to handle even the most convoluted partitioning layouts. Still no GPT support, however that's now just a matter of implementing the corresponding partition table processing code.

[Frank Mehnert]

Please reopen if the problem persists with VBox 3.2.0

[Christopher Jefferson]

I believe I am getting this issue:

~/temp$ fdisk /dev/disk0 Disk: /dev/disk0 geometry: 14751/255/63 [236978176 sectors] Signature: 0xAA55

Starting Ending

#: id cyl hd sec - cyl hd sec [ start - size]

---

1: EE 1023 254 63 - 1023 254 63 [ 1 - 409639] <Unknown ID>

*2: AF 1023 254 63 - 1023 254 63 [ 409640 - 200611705] HFS+

3: EE 1023 254 63 - 1023 254 63 [ 1 - 201023487] <Unknown ID> 4: 83 1023 254 63 - 1023 254 63 [ 201023488 - 34359296] Linux files*

~/temp$ sudo VBoxManage internalcommands createrawvmdk -filename test.vmdk -rawdisk /dev/disk0 -partitions 4 -register Oracle VM VirtualBox Command Line Management Interface Version 3.2.8 (C) 2005-2010 Oracle Corporation All rights reserved.

Two partitions start at the same place. Aborting Error reading the partition information from '/dev/disk0' The raw disk vmdk file was not created

[Klaus Espenlaub]

What you're reporting is a duplicate of #4703 and doesn't even remotely belong to this defect. If you look at the partition table then it should be obvious that any tool looking at it has no other choice than bailing out with the conclusion that there are overlapping partitions. Because all 4 are in the same place.

This will be resolved once someone finds time to work on GPT support.