[vbox-dev] VirtualBox-OSE tarball

[Klaus Espenlaub]

Till Maas wrote:
> I am resending this and hope nobody got this twice, but I cannot see my
> first attempt in my newsclient or the web archive.

Haven't seen anything - was probably swallowed by a black hole.

> Klaus Espenlaub wrote:
> 
>> Till Maas wrote:
>>> Hiyas,
>>>
>>> is it possible to get an URL to the VirtualBox-OSE tarball, that is not
>>> redirected to an URL that has (changing) GET parameters? Currently there
>>> is a h parameter, that always changes. This makes it harder to
>>> automatically fetch the tarball when I increase the version in a rpm spec
>>> file.
>> Sorry to put it so bluntly - but the only way we could possibly fix this
>> issue would be to remove this download. I'm sure no one likes that
>> option. The automatic generation of the redirection URL is due to the
>> way US export control is handled on the web server handling the
>> downloads. That's the price of reinstating the direct download links.
>> Sun is a US company and thus has to follow US legislation in this area.
> 
> I am not sure, what the US legislation demands, but I guess it would be also
> ok to put the changing part of the URL within the path instead of using GET
> parameters, e.g. make the URL look like:
> 
> http://dlc-cdn-rd.sun.com/c1/virtualbox/2.0.2/$foo/VirtualBox-2.0.2-OSE.tar.bz2

Thanks for making constructive suggestions - however we don't run the 
download servers ourselves and thus our control over how things are done 
is quite limited. We want to work on VirtualBox development and not on 
running web servers. That said, we have asked for improvements there. 
We'll see what time will bring.

> and instead of $foo, all the random data can be put.
> Btw. Red Hat (a company that also follows US legislation) provides download
> URLs that can be easily download via wget, e.g.
> http://cft.et.redhat.com/download/cft-latest.tgz

That's unfortunately an example which isn't equivalent. VirtualBox uses 
(even though it doesn't contain crypto as such) crypto code in various 
parts, and this is where life gets a little more complicated. I really 
don't want to spend much time in this context discussing US export 
regulations as these are legal matters I cannot influence.

>> It's rather interesting that in the time where there were no direct
>> download links we got less complaints about the lack of them than now
>> about this behavior which just requires another option to wget or maybe
>> 2-3 additional lines in a shell script. Nothing compared to the effort
>> we spent and still spend on having this much easier download option.
> 
> Iirc, there were always direct download links before the SUN takeover, but
> they changed often a little.

There German legislation applied. Which was much easier to deal with.

>>> Also the naming scheme of the tarball and the directory inside the
>>> tarball changes every now and then. Maybe it has settled the last
>>> releases, because I did not download every release of the 1.6.x branch.
>>> Please keep this the same. E.g. use from now on
>>> VirtualBox-$VERSION-OSE.tar.bz2.
>> That really was the case with each and every release since 1.6.0 (I
>> didn't check older releases but I think they had the same scheme). I
>> really fail to understand what the issue is.
> 
> When the URL and filename scheme is stable, on can use %{version}
> placeholders within a rpm spec, bump the version and use spectool to
> download the tarball. Also the %setup line does not need to be changed,
> unless the directory with the tarball uses a different scheme.
> 
>>> And the last favour I want to ask for, is to provide a tarball that does
>>> not contain precompiled binaries and copies of other system libs.
>> I'll comment after each suggestion.

To state the most obvious thing first which I guess I was not clear 
enough (bear with me that I'll repeat that a few times): you request us 
to spend considerable time on something which doesn't improve VirtualBox 
as such. This isn't very attractive to developers.

>>> Currently after untarring, it seems that half of the contents can be
>>> removed:
>>>
>>> 28M     tools/
>> Stripping them out will cripple the source package - some of these
>> binaries are vital for building VirtualBox, and are either modified or
>> otherwise less than trivial to get. Splitting those out per build
>> platform is a waste of time with very little benefit. I think everyone
>> would prefer us to work on more important things.
> 
> I guess I was not clear enough. I wanted to suggest to provide the tarball
> without all the binaries and local copies of libraries additionally to
> current tarball or having the removed stuff put into a separate tarball.
> Also in case of modified binaries, afaics there is no clear way to get the
> patches that are needed to build the required binaries, which is not a nice
> beheaviour within the opensource community. Even better beheaviour is to
> push the patches upstream, so that everyone can benefit from the
> modifications.

Splitting one file into two might seem trivial, but we need to do this 
(and check that everything is OK) for every release. Needs precious 
developer time and isn't really what most developers like to do.

And the "push modifications upstream" argument doesn't hold water. Too 
many projects are simply unresponsive, and even if they accept the 
modifications there is no guarantee that they haven't already introduced 
further regressions. Been there, done that. We generally try to be nice, 
but there's time constraints.

Additionally there is the delay for updates to get into ALL major 
distributions AND onto people's disks. That can easily take a couple of 
years for the sort of unusual tools we use. Building 20 tools from 
source on just slightly dated distros would make building VirtualBox 
even harder, something which we rather want to avoid.

>>> 47M     src/libs/
>> Part of this is actually a legal requirement for meeting LGPL and other
>> licensing conditions. And other parts are heavily modified libraries or
>> libraries which need to be built in exactly the provided version on
>> certain platforms. So removing this would have legal and functionality
>> implications or even break building.
> 
> I fail to understand the licensing conditions that force you to copy the
> source of other libraries into you source tree. Afaik does the LGPL not
> demand this and the every free, opensource license requires the source only
> if you provide binaries. And the modifications are better placed in patches
> sent to upstream imho.

LGPL requires the possibility of relinking. While this is possible 
without publishing the sources we prefer to be nice citizens.

And again the "push upstream" - in this place it's simply not realistic. 
We have big changes e.g. to xpcom, but I wouldn't spend a second on even 
trying to push this upstream, simply because the interests of the 
Mozilla project are clearly not the same as of the VirtualBox project. 
So this would just lead to long discussions and wasted developer time. 
Other people's priorities might be different, see below.

>>> 18M     kBuild/bin/
>> That's required to get VirtualBox building reliably. You can't mix and
>> match arbitrary kBuild tool versions and VirtualBox source trees. Also
>> adding kBuild to the list of things you have to do before even thinking
>> about building VirtualBox is something we'd rather avoid.
> 
> If you provide the kBUild stuff within a different tarball, everyone can get
> it without having to compile it. Btw. speaking about licensing conditions,
> afaics kBuild is released as GPL, which requires the availibility of the
> exact sources that were used to build the binaries. Afaics, the sources of
> the kBuild binaries are not included in the VirtualBox tarball. IANAL, but
> imho this also puts distributions at risk that mirror the tarball, because
> they can also not provide the source for these binaries.

Again - more splitting work. And the GPL argument sounds very arbitrary 
to me, as kBuild is used unmodified. Thus providing the sources is 
rather easy - just grab the matching revision off the master site.

>>> 91M     total
>> So eliminating those 91MB (out of roughly 171MB) seems attractive but in
>> fact is counter-productive. You'd get an unbuildable source tree - at
>> least following the build instructions. The list of build dependencies
>> would explode, and we'd have to write lots of additional build
>> instructions.
> 
> It would be really nice to have complete documented build dependencies, e.g.
> which version of kBuild is known to work.

Every developer in every complicated project would that this is useful 
but would point to someone else for actually doing this. :)

The "documentation" we have right now on this is in SVN, in the 
svn:externals property which pulls in the matching kBuild binaries. And 
we normally don't touch kBuild in maintenance versions, to simplify matters.

>> This might sound bad news for people following a "everything from
>> source" policy, but they can ignore those binaries and do it the hard
>> way. But we kindly ask distributions going this route to test the
> 
> The last time I tried (around version 1.6.6), every build dependency except
> of kBuild was available in Fedora and adding them to the buildrequires of
> the spec is not really hard. It only gets hard, because it is not obvious
> which special patches are applied to which files and what their upstream
> status is.

Sorry to state it so clearly - Fedora is NOT the world. VirtualBox is 
meant to be buildable on pretty much every Linux distribution, including 
somewhat dated ones - within reason.  Also I bet with Fedora you mean 
just the very latest version. How many people use that one? So far we 
tried to keep as many potential contributors in the boat as possible,

We've even spent quite some effort on making sure all the binaries we 
provide run on really every reasonable Linux distro. But that's a much 
lower effort than the one you're asking for.

Then VirtualBox is not Linux only. It's also Windows, Solaris and Mac 
OSX. The Windows user base is actually much larger - and IMHO they have 
much more reason to complain about how hard it is to build VirtualBox.

>> resulting package properly, as really anything can happen due to the
>> large deviation from the expected code base - and it would be a waste of
>> Sun developer time to even look into such problem reports. Please mark
>> such builds clearly and provide your own support contacts where the
>> users of this package can actually find them without being wizards.
>> There's nothing wrong with asking us for help after it's verified that
>> the bug is actually in the Sun-provided code base.
> 
> Imho if you expect that VirtualBox only compiles with the provided binary
> software, it is not really free and open source software.

Of course it's free and open source. All I said is that we cannot make 
any guarantees with other versions of those tools, simply because we 
cannot test them all (in all combinations). In fact for some tools we 
had to suffer lots of fix+regression combinations. What's definitely 
true is that we could do a better job providing source pointers and 
patch files. That's an area where we take shortcuts to maximize 
developer efficiency, sacrificing what you're interested in.

I think that's enough repetitions of the same tune :) I don't want to 
sound like a broken record. So time for me to get constructive.

That's why I asked for cooperation - we get a considerable number of 
bugreports that exist only in a particular third party build, and it 
takes quite some effort to debug their problem. Again time we'd love to 
spend on getting VirtualBox further.

There are other places where cooperation would lead to an improvement: 
We'd love to hear suggestions (and get help) we can put into place 
without dedicating a huge amount of precious developer time into this 
area. The more incremental such changes are the better. Simply because 
gradual changes are less intrusive to development as such, and the 
higher the chance someone (of the developer team or a contributor) will 
volunteer.

Also - we're aware that some people have better connections to the 
upstream projects than we might have. So cooperation could also mean 
some contributors either help pushing things upstream or (with feedback 
from us) establish what tools versions are known working. Talk to us, we 
know where the sore spots are.

> Just to make sure: I do not want to be disrespectful here. I enjoyed using
> my completely compiled from source virtualbox in the past and am thankful
> that you are providing the sources. The only problem is, that the source
> releases seem not to be meant to be fully free and open source, because of
> the imho big gap to free and open source best practices.

I didn't regard your mail (any of them in fact) as disrespectful. You 
certainly have valid arguments. In return I don't want to be 
disrespectful and just dismiss them with the resource argument. However 
fulfilling all your wishes is simply not possible in the near future 
with the available resources and the number of contributors we have 
right now. I'm encouraging people to contribute - and to contact us. 
We're trying to be easy to reach (time and project deadlines permitting) 
and we're answering whatever is asked.

Klaus

P.S.: This will be my last long mail in this thread, as I consider the 
main statements have been made.