sdist and preprocessors

[Ross Paterson]

On Mon, Aug 08, 2005 at 06:59:04PM -0700, Isaac Jones wrote:
> I already have a TODO item for adding a flag to "setup sdist" to
> handle this in a somewhat more fine-grained mannar:
> 
> ** if there's a flag, --include-preprocessed-sources (or something
>    better) run the preprocessing phase and include both the
>    unpreprocessed and the preprocessed sources in the source tarball?
> 
> But really, there are two kinds of preprocessors, as Ross points out.
> The kind that produce OS-independent code, and the kind that produce
> OS-dependent code.  Perhaps this concept shoudl be added to the
> PreProcessor type, and a we could have two flags to sdist:
> 
> --include-standalone-preprocessed-sources
> 
> Which would generate the OS-independent sources from tools like Alex
> and Happy... 
> 
> --include-all-preprocessed-sources
> 
> Which just includes all of the preprocessed sources as above.

Each additional option complicates the interface.  It's far better to
make the tool smart enough to choose the right thing, IMO.  So instead
of --include-standalone-preprocessed-sources, I propose:

* preprocessors are annotated with a flag saying whether they are able
  to produce system- and implementation-independent output (e.g. happy,
  alex).

* for such preprocessors, and if the preprocessor is present, sdist
  includes the preprocessed output for a CompilerFlavor of Unknown
  (e.g. happy with no options), in addition to the input.

* building a module for which preprocessor input (e.g. Foo.y) is
  present: if the preprocessor is available, use it with the specific
  CompilerFlavor (e.g. happy -g).  If not, but preprocessed output is
  available, use that.  Otherwise, fail (as now).

This will be a awkward with the current scheme of leaving preprocessed
output in the source directory, but it was already desirable to change
that to keep source directories tidy and simplify setup clean.