[RFC] Reorganizing the download directory.

[Rob Landley]

On Sunday 07 February 2010 20:57:45 Denys Vlasenko wrote:
> On Sunday 07 February 2010 01:15, Rob Landley wrote:
> > I'd like to do a couple things to http://busybox.net/downloads/
> >
> > 1) Move the actual source tarballs into a "sources" subdirectory (there's
> > a lot of them now, makes it hard to see anything else in downloads).
> >
> > I note that this change is potentially disruptive to people who download
> > via script, but it should only have to be done once.  (And we already did
> > it for the "legacy" directory with the pre-1.0 versions back when clutter
> > was getting overwhelming.)
>
> In order to not break the script, I propose just moving "sufficiently old"
> releases to legacy/

I thought about that.  The question is how to define "sufficiently old".  It also 
means that all download scripts eventually break as things get moved in 
future.

What I used to do is have everything at the top level also linked from the 
"legacy" directory so that right on the day it was released the stable URL 
would be out of the legacy directory.  (Then the links at the top level would 
last a while, but then eventually go away.)  But the legacy directory was 
always a bit of a hack, I was just trying to cut down the clutter so you could 
find what you were looking for.  It's not a good long-term solution because the 
"sufficiently old" problem is inherent and unanswerable...

Possibly if the download directory is going to accumulate enough stuff other 
than the project's source code release tarballs, then the release tarballs 
should be in their own "sources" subdirectory?  (I suggest "sources" plural 
because downloads is plural, which I always had a hard time remembering when 
"download.html" was singular.)

> > 2) Group the fixes together into a "fixes" subdirectory (so it would now
> > be fixes/1.15.2 instead of fixes-1.15.2).
>
> Same as above, move some to legacy/? I fear deeper tree only open up
> more possibilities for collecting cruft...

The project's ten years old, we've already collected cruft.  Past tense.  If 
we don't want to leave it piled in a heap, we either organize it better or we 
delete old stuff.

> I actually went ahead and moved those to legacy/, does it look ok now?

It looks fine for now, but it's a short term fix.  The oldest source tarball at 
the top level is currently march 2008.  So any download script that's a full 2 
years old (February 2008) is already broken.

It really depends whether you want to break it once in a big way and be done 
with it, or break it regularly in small ways that don't inconvenience as many 
people but which never ends...

> Another thing: we can stop generating .gz files, I think.
> It's year 2010 after all.

Except that raises the question "should we produce lzma files"?

I personally use the bz2 files, but other than ubiquitous availability the 
format hasn't really got anything to recommend it.  It's neither as efficiently 
encoded/decoded as gzip (which is why streaming protocols like ssh -C will 
continue to use gzip), or as space efficient as lzma (which does not yet have 
universal availability, and asking people to download the busybox source in a 
file format they need busybox to decompress would be silly).

If you'd like me to rewrite your bzip/bunzip code to be multi-threaded so it 
could take advantage of SMP while compressing/decompressing, then it might 
have some advantage.  (It has independent fixed-size chunks, before compression 
anyway; gz doesn't really lend itself to that so easily, and I'm not familiar 
with the guts of lzma.)  But selectively adding -lpthreads to the busybox 
makefiles really doesn't sound like fun.

In general, I tended to take my queues from the linux kernel, and they're 
still producing gz files.  But it's your call.  Back in 2003 the kernel needed 
ftp://www.kernel.org/pub/README_ABOUT_BZ2_FILES but now the main 
http://kernel.org/ page's "full source" links are straight to the bz2 files and 
nobody blinks...

> > 3) Zap Changelog.gz and the "patches" directory.  I stopped updating both
> > back during the outbreak of Bruce, and now that things have switched over
> > to git there's not much point anyway.
>
> Zapped.
>
> > 4) Is anybody still using the "qemu" directory (the one with a Red Hat 9
> > image in it for regression testing)?  As far as I know it still works
> > fine, but I dunno what our minimum acceptable development environment is
> > these days.
>
> Didn't try.

I know the image still works, I mean as far as I know the current busybox 
source still builds under it.

> > 5) Move "README" to "README-busybox" (what updates this, by the way?  Is
> > there some kind of commit hook?  Does it match what's currently in source
> > control?),
>
> It seems redundant and obsolete, ok to delete?

Look at the nav bar on the main page, under documentation, there's a link to 
that README.  It's also in the source tarball.

It's not a bad thing to have (the linux kernel source has a README too), but 
it could use a little love...

> > and add a new README describing the layout of the download directory.
>
> If layout will become non-obvious, yes.

I'd like to keep it as obvious os possible, but I'm wary of thinking that 
what's obvious to me is obvious to other people.

Also, web servers can be configured to display README file contents at the top 
of directories they provide file listings for.  (Ours currently isn't, though.)

> > 6) Add a "binaries/$VERSION" directory with prebuilt "not quite
> > allyesconfig" busybox binaries for a bunch of different targets, based on
> > the current version of the source and statically linked against uClibc. 
> > See
> > http://busybox.net/~landley/binaries/1.16.0 for what I'd put there.
>
> Wow, I like it.
> Maybe squashed one level, binaries-$VERSION?

Like busybox-armv4tl-1.16.0 you mean?  Seems a bit long, but if that's what 
you prefer, sure.  (The version number's primarily there to ensure it gets 
updated when the release does, or that if it hasn't people notice.)

As long as it's still just a postfix to the name "busybox", then

  qemu-arm ./busybox-armv4tl ls -l

Should work just fine as a quick smoke test of non-x86 bugs, which is the 
important thing.  (This bug happens for me on arm, here's how to reproduce 
it... :)

One thing I was thinking was at some point in the future I could set up a cron 
job to build nightly versions from the repository, in which case they'd need 
some way to distinguish them.  But that could go under snapshots/binaries or 
something...

Rob
-- 
Latency is more important than throughput. It's that simple. - Linus Torvalds