upload all files
This commit is contained in:
454
Modules/sqlite.include/autosetup/README.md
Normal file
454
Modules/sqlite.include/autosetup/README.md
Normal file
@@ -0,0 +1,454 @@
|
||||
Maintaining Autosetup in the SQLite Tree
|
||||
========================================================================
|
||||
|
||||
This document provides some tips and reminders for the SQLite
|
||||
developers regarding using and maintaining the [Autosetup][]-based
|
||||
build infrastructure. It is not an [Autosetup][] reference.
|
||||
|
||||
**Table of Contents**:
|
||||
|
||||
- [Autosetup API Reference](#apiref)
|
||||
- [API Tips](#apitips)
|
||||
- [Ensuring TCL Compatibility](#tclcompat)
|
||||
- [Design Conventions](#conventions)
|
||||
- Symbolic Names of Feature Flags
|
||||
- Do Not Update Global Shared State
|
||||
- [Updating Autosetup](#updating)
|
||||
- ***[Patching Autosetup for Project-local changes](#patching)***
|
||||
- [Branch-specific Customization](#branch-customization)
|
||||
|
||||
|
||||
------------------------------------------------------------------------
|
||||
|
||||
<a name="apiref"></a>
|
||||
Autosetup API Reference
|
||||
========================================================================
|
||||
|
||||
The Autosetup API is quite extensive and can be read either in
|
||||
the [files in the `autosetup` dir](/dir/autosetup) or using:
|
||||
|
||||
>
|
||||
```
|
||||
$ ./configure --reference | less
|
||||
```
|
||||
|
||||
That will include any docs from any TCL files in the `./autosetup` dir
|
||||
which contain certain (simple) markup defined by autosetup.
|
||||
|
||||
This project's own configuration-related TCL code is spread across the
|
||||
following files:
|
||||
|
||||
- [proj.tcl][]: project-agnostic utility code for autosetup-driven
|
||||
projects. This file is designed to be shared between this project,
|
||||
other projects managed under the SQLite/Hwaci umbrella
|
||||
(e.g. Fossil), and personal projects of SQLite's developers. It is
|
||||
essentially an amalgamation of a decade's worth of autosetup-related
|
||||
utility code.
|
||||
- [sqlite-config.tcl][]: utility code which is too project-specific
|
||||
for `proj.tcl`. We split this out of `auto.def` so that it can be
|
||||
used by both `auto.def` and...
|
||||
- [auto.def][]: the primary driver for the `./configure` process.
|
||||
When we talk about "the configure script," we're technically
|
||||
referring to this file, though it actually contains very little
|
||||
of the TCL code.
|
||||
- [autoconf/auto.def][]: the main driver script for the "autoconf"
|
||||
bundle's configure script. It is essentially a slightly trimmed-down
|
||||
version of the main `auto.def` file. The `autoconf` dir was ported
|
||||
from the Autotools to Autosetup in the 3.49.0 dev cycle but retains
|
||||
the "autoconf" name to minimize downstream disruption.
|
||||
|
||||
|
||||
<a name="apitips"></a>
|
||||
Autosetup API Tips
|
||||
========================================================================
|
||||
|
||||
This section briefly covers only APIs which are frequently useful in
|
||||
day-to-day maintenance and might not be immediately recognized as such
|
||||
from a casual perusal of the relevant TCL files. The complete docs of
|
||||
those with `proj-` prefix can be found in [proj.tcl][] and those with
|
||||
an `sqlite-` prefix are in [sqlite-config.tcl][]. The others are part
|
||||
of Autosetup's core packages and are scattered around [the TCL files
|
||||
in ./autosetup](/dir/autosetup).
|
||||
|
||||
In (mostly) alphabetical order:
|
||||
|
||||
- **`file-isexec filename`**\
|
||||
Should be used in place of `[file executable]`, as it will also
|
||||
check for `${filename}.exe` on Windows platforms. However, on such
|
||||
platforms it also assumes that _any_ existing file is executable.
|
||||
|
||||
- **`get-env VAR ?default?`**\
|
||||
Will fetch an "environment variable" from the first of either: (1) a
|
||||
KEY=VALUE passed to the configure script or (2) the system's
|
||||
environment variables. Not to be confused with `getenv`, which only
|
||||
does the latter and is rarely, if ever, useful in this tree.
|
||||
- **`proj-get-env VAR ?default?`**\
|
||||
Works like `get-env` but will, if that function finds no match,
|
||||
look for a file named `./.env-$VAR` and, if found, return its
|
||||
trimmed contents. This can be used, e.g., to set a developer's
|
||||
local preferences for the default `CFLAGS`.\
|
||||
Tip: adding `-O0` to `.env-CFLAGS` reduces rebuild times
|
||||
considerably at the cost of performance in `make devtest` and the
|
||||
like.
|
||||
|
||||
- **`proj-fatal msg`**\
|
||||
Emits `$msg` to stderr and exits with non-zero. Its differences from
|
||||
autosetup's `user-error` are purely cosmetic.
|
||||
|
||||
- **`proj-if-opt-truthy flag thenScript ?elseScript?`**\
|
||||
Evals `thenScript` if the given `--flag` is truthy, else it
|
||||
evals the optional `elseScript`.
|
||||
|
||||
- **`proj-indented-notice ?-error? ?-notice? msg`**\
|
||||
Breaks its `msg` argument into lines, trims them, and emits them
|
||||
with consistent indentation. Exactly how it emits depends on the
|
||||
flags passed to it (or not), as covered in its docs. This will stick
|
||||
out starkly from normal output and is intended to be used only for
|
||||
important notices.
|
||||
|
||||
- **`proj-opt-truthy flag`**\
|
||||
Returns 1 if `--flag`'s value is "truthy," i.e. one of (1, on,
|
||||
enabled, yes, true).
|
||||
|
||||
- **`proj-opt-was-provided FLAG`**\
|
||||
Returns 1 if `--FLAG` was explicitly provided to configure,
|
||||
else 0. This distinction can be used to determine, e.g., whether
|
||||
`--with-readline` was provided or whether we're searching for
|
||||
readline by default. In the former case, failure to find it should
|
||||
be treated as fatal, where in the latter case it's not.\
|
||||
Unlike most functions which deal with `--flags`, this one does not
|
||||
validate that `$FLAG` is a registered flag so will not fail fatally
|
||||
if `$FLAG` is not registered as an Autosetup option.
|
||||
|
||||
- **`proj-val-truthy value`**\
|
||||
Returns 1 if `$value` is "truthy," See `proj-opt-truthy` for the definition
|
||||
of "truthy."
|
||||
|
||||
- **`proj-warn msg`**\
|
||||
Emits `$msg` to stderr. Closely-related is autosetup's `user-notice`
|
||||
(described below).
|
||||
|
||||
- **`sqlite-add-feature-flag ?-shell? FLAG...`**\
|
||||
Adds the given feature flag to the CFLAGS which are specific to
|
||||
building libsqlite3. It's intended to be passed one or more
|
||||
`-DSQLITE_ENABLE_...`, or similar, flags. If the `-shell` flag is
|
||||
used then it also passes its arguments to
|
||||
`sqlite-add-shell-opt`. This is a no-op if `FLAG` is not provided or
|
||||
is empty.
|
||||
|
||||
- **`sqlite-add-shell-opt FLAG...`**\
|
||||
The shell-specific counterpart of `sqlite-add-feature-flag` which
|
||||
only adds the given flag(s) to the CLI-shell-specific CFLAGS.
|
||||
|
||||
- **`sqlite-configure BUILD-NAME {script}`**\
|
||||
This is where all configure `--flags` are defined for all known
|
||||
build modes ("canonical" or "autoconf"). After processing all flags,
|
||||
this function runs `$script`, which contains the build-mode-specific
|
||||
configuration bits, and then runs any finalization bits which are
|
||||
common to all build modes. The `auto.def` files are intended to contain
|
||||
exactly two commands:
|
||||
`use sqlite-config; sqlite-configure BUILD-NAME {script}`
|
||||
|
||||
- **`user-notice msg`**\
|
||||
Queues `$msg` to be sent to stderr, but does not emit it until
|
||||
either `show-notices` is called or the next time autosetup would
|
||||
output something (it internally calls `show-notices`). This can be
|
||||
used to generate warnings between a "checking for..." message and
|
||||
its resulting "yes/no/whatever" message in such a way as to not
|
||||
spoil the layout of such messages.
|
||||
|
||||
|
||||
<a name="tclcompat"></a>
|
||||
Ensuring TCL Compatibility
|
||||
========================================================================
|
||||
|
||||
One of the significant benefits of using Autosetup is that (A) this
|
||||
project uses many TCL scripts in the build process and (B) Autosetup
|
||||
comes with a TCL interpreter named [JimTCL][].
|
||||
|
||||
It is important that any TCL files used by the configure process and
|
||||
makefiles remain compatible with both [JimTCL][] and the canonical
|
||||
TCL. Though JimTCL has outstanding compatibility with canonical TCL,
|
||||
it does have a few corners with incompatibilities, e.g. regular
|
||||
expressions. If a script runs in JimTCL without using any
|
||||
JimTCL-specific features, then it's a certainty that it will run in
|
||||
canonical TCL as well. The opposite, however, is not _always_ the
|
||||
case.
|
||||
|
||||
When [`./configure`](/file/configure) is run, it goes through a
|
||||
bootstrapping process to find a suitable TCL with which to run the
|
||||
autosetup framework. The first step involves [finding or building a
|
||||
TCL shell](/file/autosetup/autosetup-find-tclsh). That will first
|
||||
search for an available `tclsh` (under several common names,
|
||||
e.g. `tclsh8.6`) before falling back to compiling the copy of
|
||||
`jimsh0.c` included in the source tree. i.e. it will prefer to use a
|
||||
system-installed TCL for running the configure script. Once it finds
|
||||
(or builds) a TCL shell, it then runs [a sanity test to ensure that
|
||||
the shell is suitable](/file/autosetup/autosetup-test-tclsh) before
|
||||
using it to run the main autosetup app.
|
||||
|
||||
There are two simple ways to ensure that running of the configure
|
||||
process uses JimTCL instead of the canonical `tclsh`, and either
|
||||
approach provides equally high assurances about configure script
|
||||
compatibility across TCL implementations:
|
||||
|
||||
1. Build on a system with no `tclsh` installed in the `$PATH`. In that
|
||||
case, the configure process will fall back to building the in-tree
|
||||
copy of JimTCL.
|
||||
|
||||
2. Manually build `./jimsh0` in the top of the checkout with:\
|
||||
`cc -o jimsh0 autosetup/jimsh0.c`\
|
||||
With that in place, the configure script will prefer to use that
|
||||
before looking for a system-level `tclsh`. Be aware, though, that
|
||||
`make distclean` will remove that file.
|
||||
|
||||
**Note that `./jimsh0` is distinctly different from the `./jimsh`**
|
||||
which gets built for code-generation purposes. The latter requires
|
||||
non-default build flags to enable features which are
|
||||
platform-dependent, most notably to make its `[file normalize]` work.
|
||||
This means, for example, that the configure script and its utility
|
||||
APIs must not use `[file normalize]`, but autosetup provides a
|
||||
TCL-only implementation of `[file-normalize]` (note the dash) for
|
||||
portable use in the configure script. Contrariwise, code-generation
|
||||
scripts invoked via `make` may use `[file normalize]`, as they'll use
|
||||
`./jimsh` or `tclsh` instead of `./jimsh0`.
|
||||
|
||||
|
||||
Known TCL Incompatibilities
|
||||
------------------------------------------------------------------------
|
||||
|
||||
A summary of known incompatibilities in JimTCL
|
||||
|
||||
- **CRNL line endings**: prior to 2025-02-05 `fconfigure -translation ...`
|
||||
was a no-op in JimTCL, and it emits CRNL line endings by default on
|
||||
Windows. Since then, it supports `-translation binary`, which is
|
||||
close enough to `-translation lf` for our purposes. When working
|
||||
with files using the `open` command, it is important to use mode
|
||||
`"rb"` or `"wb"`, as appropriate, so that the output does not get
|
||||
CRNL-mangled on Windows.
|
||||
|
||||
- **`file copy`** does not support multiple source files. See
|
||||
[](/info/61f18c96183867fe) for a workaround.
|
||||
|
||||
- **Regular expressions**:
|
||||
|
||||
- Patterns treat `\nnn` octal values as back-references (which it
|
||||
does not support). Those can be reformulated as demonstrated in
|
||||
[](/info/aeac23359bb681c0).
|
||||
|
||||
- `regsub` does not support the `\y` flag. A workaround is demonstrated
|
||||
in [](/info/c2e5dd791cce3ec4).
|
||||
|
||||
|
||||
<a name="conventions"></a>
|
||||
Design Conventions
|
||||
========================================================================
|
||||
|
||||
This section describes the motivations for the most glaring of the
|
||||
build's design decisions, in particular how they deviate from
|
||||
historical, or even widely-conventional, practices.
|
||||
|
||||
Symbolic Names of Feature Flags
|
||||
------------------------------------------------------------------------
|
||||
|
||||
Historically, the project's makefile has exclusively used
|
||||
`UPPER_UNDERSCORE` form for makefile variables. This build, however,
|
||||
primarily uses `X.y` format, where `X` is often a category label,
|
||||
e.g. `CFLAGS`, and `y` is the specific instance of that category,
|
||||
e.g. `CFLAGS.readline`.
|
||||
|
||||
When the configure script exports flags for consumption by filtered
|
||||
files, e.g. [Makefile.in][] and the generated
|
||||
`sqlite_cfg.h`, it does so in the more conventional `X_Y` form because
|
||||
those flags get exported as as C `#define`s to `sqlite_cfg.h`, where
|
||||
dots are not permitted.
|
||||
|
||||
The `X.y` convention is used in the makefiles primarily because the
|
||||
person who did the initial port finds that considerably easier on the
|
||||
eyes and fingers. In practice, the `X_Y` form of such exports is used
|
||||
exactly once in [Makefile.in][], where it's translated from `@X_Y@`
|
||||
into into `X.y` form for consumption by [Makefile.in][] and
|
||||
[main.mk][]. For example:
|
||||
|
||||
>
|
||||
```
|
||||
LDFLAGS.shobj = @SHOBJ_LDFLAGS@
|
||||
LDFLAGS.zlib = @LDFLAGS_ZLIB@
|
||||
LDFLAGS.math = @LDFLAGS_MATH@
|
||||
```
|
||||
|
||||
(That first one is defined by autosetup, and thus applies "LDFLAGS" as
|
||||
the suffix rather than the prefix. Which is more legible is a matter
|
||||
of taste, for which there is no accounting.)
|
||||
|
||||
|
||||
Do Not Update Global Shared State
|
||||
------------------------------------------------------------------------
|
||||
|
||||
In both the legacy Autotools-driven build and common Autosetup usage,
|
||||
feature tests performed by the configure script may amend global flags
|
||||
such as `LIBS`, `LDFLAGS`, and `CFLAGS`[^as-cflags]. That's
|
||||
appropriate for a makefile which builds a single deliverable, but less
|
||||
so for makefiles which produce multiple deliverables. Drawbacks of
|
||||
that approach include:
|
||||
|
||||
- It's unlikely that every single deliverable will require the same
|
||||
core set of those flags.
|
||||
- It can be difficult to determine the origin of any given change to
|
||||
that global state because those changes are hidden behind voodoo
|
||||
performed outside the immediate visibility of the configure script's
|
||||
maintainer.
|
||||
- It can force the maintainers of the configure script to place tests
|
||||
in a specific order so that the resulting flags get applied at
|
||||
the correct time and/or in the correct order.\
|
||||
(A real-life example: before the approach described below was taken
|
||||
to collecting build-time flags, the test for `-rpath` had to come
|
||||
_after_ the test for zlib because the results of the `-rpath` test
|
||||
implicitly modified global state which broke the zlib feature
|
||||
test. Because the feature tests no longer (intentionally) modify
|
||||
shared global state, that is not an issue.)
|
||||
|
||||
In this build, cases where feature tests modify global state in such a
|
||||
way that it may impact later feature tests are either (A) very
|
||||
intentionally defined to do so (e.g. the `--with-wasi-sdk` flag has
|
||||
invasive side-effects) or (B) are oversights (i.e. bugs).
|
||||
|
||||
This tree's [configure script][auto.def], [utility APIs][proj.tcl],
|
||||
[Makefile.in][], and [main.mk][] therefore strive to separate the
|
||||
results of any given feature test into its own well-defined
|
||||
variables. For example:
|
||||
|
||||
- The linker flags for zlib are exported from the configure script as
|
||||
`LDFLAGS_ZLIB`, which [Makefile.in][] and [main.mk][] then expose as
|
||||
`LDFLAGS.zlib`.
|
||||
- `CFLAGS_READLINE` (a.k.a. `CFLAGS.readline`) contains the `CFLAGS`
|
||||
needed for including `libreadline`, `libedit`, or `linenoise`, and
|
||||
`LDFLAGS_READLINE` (a.k.a. `LDFLAGS.readline`) is its link-time
|
||||
counterpart.
|
||||
|
||||
It is then up to the Makefile to apply and order the flags however is
|
||||
appropriate.
|
||||
|
||||
At the end of the configure script, the global `CFLAGS` _ideally_
|
||||
holds only flags which are either relevant to all targets or, failing
|
||||
that, will have no unintended side-effects on any targets. That said:
|
||||
clients frequently pass custom `CFLAGS` to `./configure` or `make` to
|
||||
set library-level feature toggles, e.g. `-DSQLITE_OMIT_FOO`, in which
|
||||
case there is no practical way to avoid "polluting" the builds of
|
||||
arbitrary makefile targets with those. _C'est la vie._
|
||||
|
||||
|
||||
[^as-cflags]: But see this article for a detailed discussion of how
|
||||
autosetup currently deals specifically with CFLAGS:
|
||||
<https://msteveb.github.io/autosetup/articles/handling-cflags/>
|
||||
|
||||
|
||||
<a name="updating"></a>
|
||||
Updating Autosetup
|
||||
========================================================================
|
||||
|
||||
Updating autosetup is, more often than not, painless. It requires having
|
||||
a checked-out copy of [the autosetup git repository][autosetup-git]:
|
||||
|
||||
>
|
||||
```
|
||||
$ git clone https://github.com/msteveb/autosetup
|
||||
$ cd autosetup
|
||||
# Or, if it's already checked out:
|
||||
$ git pull
|
||||
```
|
||||
|
||||
Then, from the top-most directory of an SQLite checkout:
|
||||
|
||||
>
|
||||
```
|
||||
$ /path/to/autosetup-checkout/autosetup --install .
|
||||
$ fossil status # show the modified files
|
||||
```
|
||||
|
||||
Unless the upgrade made any incompatible changes (which is exceedingly
|
||||
rare), that's all there is to it. After that's done, **apply a patch
|
||||
for the change described in the following section**, test the
|
||||
configure process, and check it in.
|
||||
|
||||
<a name="patching"></a>
|
||||
Patching Autosetup for Project-local Changes
|
||||
------------------------------------------------------------------------
|
||||
|
||||
Autosetup reserves the flag name **`--debug`** for its own purposes,
|
||||
and its own special handling of `--enable-...` flags makes `--debug`
|
||||
an alias for `--enable-debug`. As this project has a long history of
|
||||
using `--enable-debug`, we patch autosetup to use the name
|
||||
`--autosetup-debug` in place of `--debug`. That requires (as of this
|
||||
writing) four small edits in
|
||||
[/autosetup/autosetup](/file/autosetup/autosetup), as demonstrated in
|
||||
[check-in 3296c8d3](/info/3296c8d3).
|
||||
|
||||
If autosetup is upgraded and this patch is _not_ applied the invoking
|
||||
`./configure` will fail loudly because of the declaration of the
|
||||
`debug` flag in `auto.def` - duplicated flags are not permitted.
|
||||
|
||||
<a name="branch-customization"></a>
|
||||
Branch-specific Customization
|
||||
========================================================================
|
||||
|
||||
Certain vendor-specific branches require slight configure script
|
||||
customization. Rather than editing `sqlite-config.tcl` for this,
|
||||
which frequently leads to merge conflicts, the following approach
|
||||
is recommended:
|
||||
|
||||
In the vendor-specific branch, create a file named
|
||||
`autosetup/sqlite-custom.tcl`.
|
||||
|
||||
That file should contain the following content...
|
||||
|
||||
If flag customization is required, add:
|
||||
|
||||
>
|
||||
```
|
||||
proc sqlite-custom-flags {} {
|
||||
# If any existing --flags require different default values
|
||||
# then call:
|
||||
options-defaults {
|
||||
flag-name new-default-value
|
||||
...
|
||||
}
|
||||
# ^^^ That will replace the default value but will not update
|
||||
# the --help text, which may lead to some confusion:
|
||||
# https://github.com/msteveb/autosetup/issues/77
|
||||
|
||||
return {
|
||||
{*} {
|
||||
new-flag-name => {Help text}
|
||||
...
|
||||
}
|
||||
}; #see below
|
||||
}
|
||||
```
|
||||
|
||||
That function must return either an empty string or a list in the form
|
||||
used internally by [sqlite-config.tcl][]'s `sqlite-configure`.
|
||||
|
||||
Next, define:
|
||||
|
||||
>
|
||||
```
|
||||
proc sqlite-custom-handle-flags {} {
|
||||
... do any custom flag handling here ...
|
||||
}
|
||||
```
|
||||
|
||||
That function, if defined, will be called relatively late in the
|
||||
configure process, before any filtered files are generated but after
|
||||
all other significant processing.
|
||||
|
||||
|
||||
[Autosetup]: https://msteveb.github.io/autosetup/
|
||||
[auto.def]: /file/auto.def
|
||||
[autoconf/auto.def]: /file/autoconf/auto.def
|
||||
[autosetup-git]: https://github.com/msteveb/autosetup
|
||||
[proj.tcl]: /file/autosetup/proj.tcl
|
||||
[sqlite-config.tcl]: /file/autosetup/sqlite-config.tcl
|
||||
[Makefile.in]: /file/Makefile.in
|
||||
[main.mk]: /file/main.mk
|
||||
[JimTCL]: https://jim.tcl.tk
|
||||
Reference in New Issue
Block a user