Build system idea

[John Meacham]

On Wed, Aug 27, 2008 at 10:18:59PM +0100, Duncan Coutts wrote:
> On Wed, 2008-08-27 at 06:13 -0700, John Meacham wrote:
> > The problem with the way cabal wants to mix with make/autoconf is that
> > it is the wrong way round. make is very good at managing pre-processors,
> > dependency tracking and calling external programs in the right order, in
> > parallel, and as needed. cabal is generally good at building a single
> > library or executable given relatively straightforward haskell source.
> > (I know it _can_ do more, but this is mainly what it is good at).
> >
> > The way this should work is that make determines what haskell libraries
> > need to be built, and what haskell files need to be generated to allow
> > cabal to run and calls cabal to build just the ones needed. cabal as a
> > build tool that make calls is much more flexible and in tune with each
> > tools capabilities.
>
> I'd say if you're using make for all that, then use it to build the
> haskell modules too. That gives the advantage of incremental and
> parallel builds, which Cabal does not do yet (though we've got a GSoC
> project just coming to an end which does this).

So, don't use cabal at all? that is the solution I have been going with so
far and am trying to remedy.

> > The other issue is with cabal files themselves which are somewhat
> > conflicted in purpose. on one hand, you have declarative stuff about a
> > package. name, version, etc... information you want before you start to
> > build something. but then you have build-depends, which is something
> > that you cannot know until after your configuration manager (whatever it
> > may be, autoconf being a popular one) is run.
>
> Ah, but that's where the autoconf and Cabal models part ways.
>
> > What packages you depend on are going to depend on things like what
> > compiler you have installed, your configuration options, which
> > packages are installed, what operating system you are running on,
> > which kernel version you are running, which c libraries you have
> > installed. etc. things that cannot be predicted before the
> > configuration is actually run.
>
> So Cabal takes the view that the relationship between features and
> dependencies should be declarative. autoconf is essentially a function
> from a platform environment to maybe a configuration. That's a very
> flexible approach, the function is opaque and can do whatever feature
> tests it likes. The downside is that it is not possible to work out what
> the dependencies are. It might be able to if autoconf explained the
> result of its decisions, but even then, it's not possible to work out
> what dependencies are required to get a particular feature enabled. With
> the Cabal approach these things are explicit.

unfortunately the cabal approach doesn't work. note, I am not saying a
declarative configuration manager won't work. in fact, I have sketched a
design for one on occasion. but cabal's particular choices are broken.
It is treading the same waters that made 'imake' fail.

the ideas of forwards and backwards compatability are _the_ defining
features of a configuration manager. Think about this, I can take my old
sunsite CD, burned _ten years_ ago and take the unchanged tarballs off
that CD and ./configure && make and in general most will work. many were
written before linux even existed, many were written with non gcc
compilers, yet they work today. The cabal way wasn't able to handle a
single release of ghc and keep forwards or backwards compatability.

That any project ever had to be changed to use the flag 'split-base' is
a travesty. What about all the projects on burnt cds or that don't have
someone to update them? 20 years from now when we are all using 'fhc'
(Fred's Haskell Compiler) will we still have this reference to
'split-base' in our cabal files? how many more flags will have
accumulated by then? Sure it's declarative, but in a language that
doesn't make sense without the rule-book.  autoconf tests things like
'does a library named foo exist and export bar'. 'is char signed or
unsigned on the target system'. those are declarative statement and
have a defined meaning through all time. (though, implemented in a
pretty ugly imperative way) That is what allows autoconfed packages to
be compiled by compilers on systems that were never dreamed of when the
packages were written.


> The conditionals in a .cabal file can be read in either direction so it
> is possible for a package manager to automatically work out what deps
> would be needed for that optional libcurl feature, or GUI.


In the cabal framework Will cabal be able to do things like cross
compile a c file to an object file, and deconstruct the generated ELF
file to determine parameters needed for an unknown embedded platform
_and_ do so without me requiring the user to upgrade their cabal? This
is an example of the type of autoconf test that comes up in the real
world. You can never come up with a language that will have every needed
primitive, any restricted set will ultimately not be enough for someone.
and the only alternative is pretty much to not use cabal at all or hack
around it in odd ways.


> The other principle is that the packager, the environment is in control
> over what things the package 'sees'. With autoconf, the script can take
> into account anything it likes, even if you'd rather it did not. Eg it's
> important to be able to build a package that does not have that optional
> dependency, even though the C lib is indeed installed on the build
> machine, because I may be configuring it for a machine without the C
> lib. Sure, some good packages allow those automagic decisions to be
> overridden, but many don't and of course there is no easy way to tell if
> it's picking up deps it should not. So one of the principles in Cabal
> configuration is that all decisions about how to configure the package
> are transparent to the packager and can be overridden.

I am not sure what you mean by this. autoconf's flexibility in this
regard is pretty exceptional when written properly. Native
cross-compilation is one of autoconfs strengths and a big motivating
factor in its design.

> Now currently, Cabal only has a partial implementation of the concept
> because when it tries to find a configuration that works in the current
> environment (which it only does if the configuration is not already
> fully specified by the packager) it only considers dependencies on
> haskell packages. Obviously there are a range of other dependencies
> specified in the .cabal file and it should use them all, in particular
> external C libs.

And there are many other possible implementations of configuration
managers. I fully believe that the next big one will come out of the
haskell community, we are a good bunch of people. But it won't if
innovation is stifled by cabal _insisting_ on using its own
configuration manager and cabal being promoted as 'the way' to do
things.

This is completely independent of my opinions of cabal as a
configuration manager, I would just hate to see such an enticing area of
research be cut off prematurely. 

If cabal is going to be the way to do things with haskell, that means it
cannot be the place to try out ones own pet projects about how one
thinks things should be. A declarative configuration manager is an
intruiging project. one I want to see people work on in different
directions, but it is new research.


> So I accept that we do not yet cover the range of configuration choices
> that are needed by the more complex packages (cf darcs), but I think
> that we can and that the approach is basically sound. The fact that we
> can automatically generate distro packages for hundreds of packages is
> not insignificant. This is just not possible with the autoconf approach.

This is just utterly untrue. autoconfed packages that generate rpms,
debs, etc are quite common. The only reason cabal can autogenerate
distro packages for so many is that many interesting or hard ones just
_arn't possible with cabal at all_. Cabal's inflexibility puts a huge
selection bias on the population of cabalized programs.

> > Then you have cabal as a packaging system (or perhaps hackage/cabal
> > considered together). Which has its own warts, if it is meant to live in
> > the niche of package managers such as rpm or deb, where are the
> > 'release' version numbers that rpms and debs have for one example? If it is
> > meant to be a tarball like format, where is the distinction between
> > 'distribution' and 'source' tarballs?
>
> Right, it's supposed to be the upstream release format, tarballs. Distro
> packages obviously have their additional revision numbers.

one might say hackage is a distro in and of itself, so should have
similar numbers. reusing the same file directly for the packager and the
build system makes things like this trickier than they need to be.

> > For instance, jhc from darcs for developers requires
> > perl,ghc,DrIFT,pandoc,autotools, and happy.  however the jhc
> > tarball requires _only_ ghc. nothing else. This is because the make
> > dist target is more interesting than just taring up the source. (and
> > posthooks/prehooks don't really help. they are sort of equivalent to
> > saying 'write your own build system'.)
>
> Right. Cabal does that too (or strictly speaking, the Simple build
> system can do this). For pre-processors that are platform independent
> (like alex, happy etc) it puts the pre-processed source into the release
> tarball. It's also possible to make tarballs without the pre-generated
> files if it's important.


Sort of. but cabal can only do these things because they are _built in_
to cabal. make will happily use DrIFT, figure out dependencies for ghc,
gcc, and jhc, and build my rpms without having to be modified itself.
Because it was designed that way.

> > One of the biggest sources of conflict arise from using cabal as a
> > configuration manager. A configuration managers entire purpose is to
> > examine the system and figure out how to adapt your programs build to
> > the system.
>
> Well, that's the autoconf view. It's not the only way of looking at it
> as I explained above (perhaps not very clearly). I'd say a configuration
> manager should negotiate between the package and the
> packager/user/environment to find a configuration that is satisfactory
> to all (which requires information flow in both directions).
>
> > this is completely 100% at odds with the idea of users
> > having to 'upgrade' cabal. Figuring out how to adapt your build to
> > whatever cabal is installed or failing gracefully if you can't is
> > exactly the job of the configuration manager. something like autoconf.
> > This is why _users_ need not install autoconf, just developers. since
> > autoconf generates a portable script is so that users are never told to
> > upgrade their autoconf. if a developer wants to use new features, he
> > gets the new autoconf and reruns 'autoreconf'. The user is never
> > asked to update anything that isn't actually needed for the project
> > itself. This distinction is key fora configuration manager and really
> > conflicts with cabal wanting to also be a build system and package
> > manager. It is also what is needed for forwards and backwards
> > compatibility.
>
> I suppose in principle it'd be possible to ship the build system in
> every package like autoconf/automake does. Perhaps we should allow that
> as an option. It's doable since the Setup.hs can import local modules.

I don't see what you mean, autoconf doesn't "ship the build system" with
the package any more than ghc ships ghc with every binary it produces.

autoconf is a _compiler_ of a domain specific language to a portable
intermediate language by design. This means that autoconf need not be
upgraded or installed by users, yet developers are free to take
advantage of autoconf's newest features without troubling their users
because what is distributed is autoconfs compiled output.

If a user has to upgrade their cabal to install a package, then cabal is
broken as a configuration manager by design. If I were willing to make a
user upgrade their system, there would be _no need_ for a configuration
manager at all. The problem of building the most recent and updated
library with the most recent and updated compiler on a fully up to date
system is a _non problem_.


> > All in all, I think these conflicting goals of cabal make it hard to use
> > in projects and have led to very odd design choices. I think external
> > tools should not be the exception but rather the rule. Not that cabal
> > shouldn't come with a full set of said tools. But as long as they are
> > integrated I don't see cabal's design problems being fixed, meerly
> > augmented with various work-arounds.
>
> One issue, with a pick and mix approach is what is the top level
> interface that users/package managers use? The current choice (which I'm
> not at all sure is the right one) is a Setup.hs file that imports its
> build system from a library that's already on the system (or a custom
> one implemented locally). So a system that uses make underneath still
> has to present the Setup.hs interface so that package managers can use
> it in a uniform way. You mention at the top that you think the
> make/cabal relationship is the wrong way round, but the Cabal/Setup.hs
> interface has to be the top level one (at least at the moment) so you'd
> have Setup.hs call make and make call it back again to build various
> bits like libs etc?

Right now I just have ./configure && make be the way to build things,
and the ./configure generates an appropriate cabal file when needed. But
the 'cabal proxy' stub cabal file similar to what you are saying is also
something I have considered (only for haskell libraries I want to put on
hackage) but it is far from ideal. 

As for programs written in haskell, I don't want people's first
impression of haskell being "oh crap, I gotta learn a new way to build
things just because this program is written in some odd language called
'haskell'" I don't care how awesome a language is, I am going to be
annoyed by having to deal with it when I just want to compile/install a
program. It will leave a bad taste in my mouth. I would much rather
peoples first impression be "oh wow, this program is pretty sweet. I
wonder what it is written in?" hence they all use ./configure && make by
design rather than necessity.

> Do you think that separating the Simple build system from the
> declarative part of Cabal would help? It'd make it more obvious that the
> build system part really is replaceable which currently is not so
> obvious since they're in the same package. I'm not averse to splitting
> them if it'd help. They're already completely partitioned internally.

Yes it would help signifigantly if it were its own program, invoked by
cabal just like hmake or make or mk or cook or bake would be. It would
be a step in the right direction. But what I'd really like to see is a
split of the configuration management from the parts that meerly
describe the package.

I sometimes hear that I just shouldn't use cabal for some projects but,
when it comes down to it. If cabal is a limited build/configuration
system in any way, why would I ever choose it when starting a project
when I know it is either putting a limit on my projects ability to
innovate or knowing that at some point in the future I am going to have
to switch build systems?

If cabal isn't suitable or convinient for some projects (which we all
admit) and cabal is the haskell way of doing things then the perception
will be that _haskell_ is not suitable for said projects. And that is
what I fear.

        John

-- 
John Meacham - ⑆repetae.net⑆john⑈