This rewrites the HTML rendering pipeline to use a tree data structure,
and implements a custom HTML serializer. The intent is to make it easier
to make changes and to manipulate the output. This should make some
future changes much easier.
This is a large change, but I'll try to briefly summarize what's
changing:
- All of the HTML rendering support has been moved out of
mdbook-markdown into mdbook-html. For now, all of the API surface is
private, though we may consider ways to safely expose it in the
future.
- Instead of using pulldown-cmark's html serializer, this takes the
pulldown-cmark events and translates them into a tree data structure
(using the ego-tree crate to define the tree). See `tree.rs`.
- HTML in the markdown document is parsed using html5ever, and then
lives inside the same tree data structure. See `tokenizer.rs`.
- Transformations are then applied to the tree data structure. For
example, adding header links or hiding code lines.
- Serialization is a simple process of writing out the nodes to a
string. See `serialize.rs`.
- The search indexer works on the tree structure instead of re-rendering
every chapter twice. See `html_handlebars/search.rs`.
- The print page now takes a very different approach of taking the
same tree structure built for rendering the chapters, and applies
transformations to it. This avoid re-parsing everything again. See
`print.rs`.
- I changed the linking behavior so that links on the print page
link to items on the print page instead of outside the print page.
- There are a variety of small changes to how it serializes as can be
seen in the changes to the tests. Some highlights:
- Code blocks no longer have a second layer of `<pre>` tags wrapping
it.
- Fixed a minor issue where a rust code block with a specific
edition was having the wrong classes when there was a default
edition.
- Drops the ammonia dependency, which significantly reduces the number
of dependencies. It was only being used for a very minor task, and
we can handle it much more easily now.
- Drops `pretty_assertions`, they are no longer used (mostly being
migrated to the testsuite).
There's obviously a lot of risk trying to parse everything to such a low
level, but I think the benefits are worth it. Also, the API isn't super
ergonomic compared to say javascript (there are no selectors), but it
works well enough so far.
I have not run this through rigorous benchmarking, but it does have a
very noticeable performance improvement, especially in a debug build.
I expect in the future that we'll want to expose some kind of
integration with extensions so they have access to this tree structure
(or some kind of tree structure).
Closes https://github.com/rust-lang/mdBook/issues/1736
This adds a bunch of tests to better exercise the HTML rendering and to
be able to track any changes in its behavior.
This includes a new `check_all_main_files` to more conveniently check
the HTML content of every chapter in a book.
This is a very simplistic utility to compare the output of different
versions of mdbook. This is useful when making changes to compare
real-world books to see what changes actually happen.
This is a variation of scripts that I have been using for a few years.
This could definitely use some improvements, but this seems like it
could be useful as-is.
This changes partition_source so that instead of allocating new strings,
it just returns slices into the original string. It probably doesn't
make a big difference perf-wise, but I felt more comfortable with this,
and also felt it was a little easier to understand exactly what it was
doing.
This is generally equivalent except for the possibility of not having a
newline at the end. In practice that doesn't matter because markdown
code blocks always have a newline. However, to be defensive, the caller
will check for this.
This adds the `ToUrlPath` helper trait to convert a Path to a path
suitable for use in HTML (replacing `normalize_path`).
This also fixes a minor bug where on Windows the next/prev links were
using a double forward slash. I don't think this is possible, since
chapter links are derived from the summary, but I'm noting just in case.
It's also not too much of an issue since double slashes are normally
just treated as a single.
This adds the `Book::chapters` iterator (and `for_each_chapter_mut`) to
iterate over non-draft chapters. This is a common pattern I keep
encountering, and I figure it might simplify things. It runs a little
risk that callers may not be properly handling every item type, but I
think it should be ok.
This removes the macro_use for clap just because I'm not a big fan of
glob-style imports like this. I think being a little more explicit here
makes it a little clearer where these macros come from.
This switches to using the tracing crate instead of log. Tracing
provides a lot of nice features which we can take advantage of moving
forward.
This also adjusts the output fairly significantly. This includes:
- Switched the environment variable from RUST_LOG to MDBOOK_LOG.
- Dropped the timestamp. I experimented with various different time
displays, but ultimately decided to omit it for now. I don't think
I've ever found it to be useful, and it takes up a very significant
amount of space. It could potentially be useful for basic profiling,
but I think there are other, better mechanisms for that. We could
consider leveraging tracing itself for doing some basic profiling
(like using something like tracing-chrome).
- Dropped the target unless MDBOOK_LOG is set. The target tends to be
pretty noisy, and doesn't really convey much information unless you
are debugging or otherwise trying to adjust the log output.
- Added color.
- Slightly reworked the way the error cause trace is displayed.
- Slightly changed the way html5ever filtering is done, as well as add
handlebars to the list since they both are very noisy. You can
override this now by explicitly listing them as targets.
I still expect that mdbook will eventually change how it displays things
to the console, possibly switching away from tracing and printing things
itself. However, that is a larger project for the future.
This changes the publishing process so that when publishing the guide
and the current version is a pre-release, it will be pushed to a
directory called `/pre-release/`.
This also switches from using simpleinfra's SSH-based script to a simple
push using normal git commands.
This uses the new guide-helper preprocessor to insert the version string
on the continuous integration guide page. This should make it easier to
bump new versions.
This adds a test to ensure that the interface for preprocessors and
renderers does not change unexpectedly, particularly in a semver
compatible release.
Closes https://github.com/rust-lang/mdBook/issues/1574
This removes the `non_exhaustive` attribute from the `Book` and its
inner types `BookItem` and `Chapter`. These were added in
https://github.com/rust-lang/mdBook/pull/2779. After thinking about it
more, I realized that these types cannot be extended in a
semver-compatible way, so I am fine with allowing them be exhaustive.
The problem is that with CmdPreprocessor, the `Book` will be
re-serialized by a preprocessor, which could potentially be on an older
version. Attempting to add any new fields/variants means that either the
deserialization will fail, or the new fields will be stripped by the
preprocessor.
These could potentially be structured such that they have a
`serde(flatten)` or Other/Unknown variant so that a preprocessor would
at least see the extra fields/variants and pass them along back to the
output. However, a preprocessor or renderer wouldn't know what to do
with those new fields/variants (particularly `BookItem`) which would
itself be a problem. It's still possible to do something like this in
the future, but for now I think it's fine to restrict these to
semver-major changes.
This test is failing on CI. I don't know why, as I cannot reproduce
locally. Perhaps it is due to the update to browser-ui-test? Or perhaps
some events from the new nav bar are delaying something?
This adds dynamic navigation of headers of the current page in the
sidebar. This is intended to help the user see what is on the current
page, and to be able to more easily navigate it. The "current" header is
tracked based on the scrolling behavior of the user, and is marked with
a small circle. This includes automatic folding to help keep it from
being too unwieldy on a page with a lot of nested headers.
This includes the `output.html.sidebar-header-nav` option to disable it.
I'm sure there are tweaks, fixes, and improvements that can be made. I'd
like to get this out now, and iterate on it over time to make
improvements.
This adds the ability to pass options to browser-ui-test, which can help
with debugging or doing things like snapshot work. It's maybe not the
cleanest since it doesn't support space-separated args, but should be
good enough.
This enables the hash-files setting by default. We have been running it
for a while, and it seems most of the issues have been resolved. This
should help with more reliably loading content like the toc contents.
This is helpful for matching patterns within a larger file. The error
message isn't quite as good, since it doesn't explicitly say "pattern
not found", but I think you can figure it out from the context.
This sprinkles track_caller on some more test functions to give more
useful line numbers on errors when a test fails.
read_to_string was changed since it couldn't track caller on a closure.
This updates eslint to lint on the toc.js.hbs file. This file uses a
small amount of hbs templating which eslint can't handle. This adds a
hacky preprocessor which will strips out the handlebars tags so that the
file can be linted.
