The marshaling functions for pandoc's AST are extracted into a separate
package. The package comes with a number of changes:
- Pandoc's List module was rewritten in C, thereby improving error
messages.
- Lists of `Block` and `Inline` elements are marshaled using the new
list types `Blocks` and `Inlines`, respectively. These types
currently behave identical to the generic List type, but give better
error messages. This also opens up the possibility of adding
element-specific methods to these lists in the future.
- Elements of type `MetaValue` are no longer pushed as values which
have `.t` and `.tag` properties. This was already true for
`MetaString` and `MetaBool` values, which are still marshaled as Lua
strings and booleans, respectively. Affected values:
+ `MetaBlocks` values are marshaled as a `Blocks` list;
+ `MetaInlines` values are marshaled as a `Inlines` list;
+ `MetaList` values are marshaled as a generic pandoc `List`s.
+ `MetaMap` values are marshaled as plain tables and no longer
given any metatable.
- The test suite for marshaled objects and their constructors has
been extended and improved.
- A bug in Citation objects, where setting a citation's suffix
modified it's prefix, has been fixed.
* Drop old windows 32-bit constraints
- basement >= 0.0.10 was 0.0.12 on stackage-18.10
- foundation >= 0.0.23 was 0.0.26.1 on stackage-18.10
* Update cabal `tested-with` field to correspond to `ci.yml` matrix
* ghc: 8.10.{2,4} → 8.10.7
This allows us to get rid of the old custom prelude and
some crufty cpp. But the primary reason for this is that
conduit has bumped its base lower bound to 4.12, making it
impossible for us to support lower base versions.
This is better as an example. And it is faster than pandoc's
regular creole parser, which shows that high-performance readers
can be developed this way.
New module Text.Pandoc.Readers.Custom, exporting
readCustom [API change].
Users can now do `-f myreader.lua` and pandoc will treat the
script myreader.lua as a custom reader, which parses an input
string to a pandoc AST, using the pandoc module defined for
Lua filters.
A sample custom reader can be found in data/reader.lua.
Closes#7669.
Compiles the 'lpeg' library (Parsing Expression Grammars For Lua) into
the program.
Package maintainers may choose to rely on package dependencies to make
lpeg available, in which case they can compile the with the constraint
`lpeg +rely-on-shared-lpeg-library`.
This fixes issues with
- misleading error messages when a required function parameter is
omitted;
- absent properties still being listed in the output of `pairs`; and
- alias accessing leading to errors instead of returning `nil`, e.g.
with `(pandoc.Str '').identifier`.
Fixes: #7661
See also: #7657
Reasons:
- Performance: HsYAML is around 20 times slower in parsing
large YAML bibliographies (#6084).
- An issue was submitted to HsYAML, but it hasn't gotten
any attention. HsYAML seems borderline unmaintained; it hasn't
had a commit in over a year.
- Unfortunately this goes back on our attempts to free ourselves
from C dependencies (#4535). But I don't see a better alternative
until a better pure Haskell parser is available.
Closes#6084.
Notes:
- We've removed the FromYAML instances for all types that had
them, since this is a HsYAML-specific typeclass [API change].
(The yaml package just uses From/ToJSON.)
- Unlike HsYAML (in the configuration we were using), yaml
parses 'Y', 'N', 'Yes', 'No', 'On', 'Off' as boolean values.
Users may need to quote these when they are meant to be
interpreted as strings. Similarly, 'null' is parsed as
a YAML null value (and will be treated as an empty string
by pandoc rather than the string 'null'). Quoting it will
force it to be interpreted as a string.
- Some tests had to be adjusted accordingly.
- Pandoc now behaves better when the YAML metadata contains
escaping errors: instead of just falling back on treating
the section as a table, it raises a YAML parsing error.
- Adds a new `pandoc.AttributeList()` constructor, which creates the
associative attribute list that is used as the third component of
`Attr` values. Values of this type can often be passed to constructors
instead of `Attr` values.
- `AttributeList` values can no longer be indexed numerically.
The new HsLua version takes a somewhat different approach to marshalling
and unmarshalling, relying less on typeclasses and more on specialized
types. This allows for better performance and improved error messages.
Furthermore, new abstractions allow to document the code and exposed
functions.
In PowerPoint, the content of a top-level list is at the same level as
the content of a top-level paragraph – the only difference is that a
list style has been applied.
At the moment, the pptx writer increments the paragraph level on each
list, turning what should be top-level lists into second-level lists.
This commit changes that logic, only incrementing the paragraph level on
continuation paragraphs of lists.
- Fixes https://github.com/jgm/pandoc/issues/4828
- Fixes https://github.com/jgm/pandoc/issues/4663
Update tests.
Reason: it turns out that the native output generated by
pretty-simple isn't always readable by the native reader.
According to https://github.com/cdepillabout/pretty-simple/issues/99
it is not a design goal of the library that the rendered values
be readable using 'read'. This makes it unsuitable for our
purposes.
pretty-show is a bit slower and it uses 4-space indents
(non-configurable), but it doesn't have this serious drawback.
Previously we used our own homespun formatting. But this
produces over-long lines that aren't ideal for diffs in tests.
Easier to use something off-the-shelf and standard.
Closes#7580.
Performance is slower by about a factor of 10, but this isn't
really a problem because native isn't suitable as a serialization
format. (For serialization you should use json, because the reader
is so much faster than native.)
In PowerPoint, it’s possible to specify footers across all slides,
containing a date (optionally automatically updated to today’s date),
the slide number (optionally starting from a higher number than 1), and
static text. There’s also an option to hide the footer on the title
slide.
Before this commit, none of that footer content was pulled through from
the reference doc: this commit supports all the functionality listed
above.
There is one behaviour which may not be immediately obvious: if the
reference doc specifies a fixed date (i.e. not automatically updating),
and there’s a date specified in the metadata for the document, the
footer date is replaced by the metadata date.
- Include date, slide number, and static footer content from reference
doc
- Respect “slide number starts from” option
- Respect “Don’t show on title slide” option
- Add tests
In the reveal-js output, it’s possible to use reveal’s
`data-background-image` class on a slide’s title to specify a background
image for the slide.
With this commit, it’s possible to use `background-image` in the same
way for pptx output. Only the “stretch” mode is supported, and the
background image is centred around the slide in the image’s larger axis,
matching the observed default behaviour of PowerPoint.
- Support `background-image` per slide.
- Add tests.
- Update manual.
- Support -i option
- Support incremental/noincremental divs
- Support older block quote syntax
- Add tests
One thing not clear from the manual is what should happen when the input
uses a combination of these things. For example, what should the
following produce?
```md
::: {.incremental .nonincremental}
- are
- these
- incremental?
:::
::: incremental
::::: nonincremental
- or
- these?
:::::
:::
::: nonincremental
> - how
> - about
> - these?
:::
```
In this commit I’ve taken the following approach, matching the observed
behaviour for beamer and reveal.js output:
- if a div with both classes, incremental wins
- the innermost incremental/nonincremental div is the one which takes
effect
- a block quote containing a list as its first element inverts whether
the list is incremental, whether or not the quote is inside an
incremental/non-incremental div
I’ve added some tests to verify this behaviour.
This commit closes issue #5689
(https://github.com/jgm/pandoc/issues/5689).
There was a mistake in the logic used to choose between the Comparison
and Two Content layouts: if one column contained only non-text (an image
or a table) and the other contained only text, the Comparison layout was
chosen instead of the desired Two Content layout.
This commit fixes that logic:
> If either column contains text followed by non-text, use Comparison.
Otherwise, use Two Content.
It also adds a test asserting this behaviour.
Until now, the pptx writer only supported four slide layouts: “Title
Slide” (used for the automatically generated metadata slide), “Section
Header” (used for headings above the slide level), “Two Column” (used
when there’s a columns div containing at least two column divs), and
“Title and Content” (used for all other slides).
This commit adds support for three more layouts: Comparison, Content
with Caption, and Blank.
- Support “Comparison” slide layout
This layout is used when a slide contains at least two columns, at
least one of which contains some text followed by some non-text (e.g.
an image or table). The text in each column is inserted into the
“body” placeholder for that column, and the non-text is inserted into
the ObjType placeholder. Any extra content after the non-text is
overlaid on top of the preceding content, rather than dropping it
completely (as currently happens for the two-column layout).
+ Accept straightforward test changes
Adding the new layout means the “-deleted-layouts” tests have an
additional layout added to the master and master rels.
+ Add new tests for the comparison layout
+ Add new tests to pandoc.cabal
- Support “Content with Caption” slide layout
This layout is used when a slide’s body contains some text, followed by
non-text (e.g. and image or a table). Before now, in this case the image
or table would break onto a new slide: to get that output again, users
can add a horizontal rule before the image or table.
+ Accept straightforward tests
The “-deleted-layouts” tests all have an extra layout and relationship
in the master for the Content with Caption layout.
+ Accept remove-empty-slides test
Empty slides are still removed, but the Content with Caption layout is
now used.
+ Change slide-level-0/h1-h2-with-text description
This test now triggers the content with caption layout, giving a
different (but still correct) result.
+ Add new tests for the new layout
+ Add new tests to the cabal file
- Support “Blank” slide layout
This layout is used when a slide contains only blank content (e.g.
non-breaking spaces). No content is inserted into any placeholders in
the layout.
Fixes#5097.
+ Accept straightforward test changes
Blank layout now copied over from reference doc as well, when
layouts have been deleted.
+ Add some new tests
A slide should use the blank layout if:
- It contains only speaker notes
- It contains only an empty heading with a body of nbsps
- It contains only a heading containing only nbsps
- Change ContentType -> Placeholder
This type was starting to have a constructor for each placeholder on
each slide (e.g. `ComparisonUpperLeftContent`). I’ve changed it
instead to identify a placeholder by type and index, as I think that’s
clearer and less redundant.
- Describe layout-choosing logic in manual
- Use dashes consistently rather than underscores
- Make a folder for each set of tests
- List test files explicitly (Cabal doesn’t support ** until version
2.4)
+ Add sandbox feature for readers. When this option is used,
readers and writers only have access to input files (and
other files specified directly on command line). This restriction
is enforced in the type system.
+ Filters, PDF production, custom writers are unaffected. This
feature only insulates the actual readers and writers, not
the pipeline around them in Text.Pandoc.App.
+ Note that when `--sandboxed` is specified, readers won't have
access to the resource path, nor will anything have access to
the user data directory.
+ Add module Text.Pandoc.Class.Sandbox, defining
`sandbox`. Exported via Text.Pandoc.Class. [API change]
Closes#5045.
before passing them off to citeproc.
This ensures that we get proper localization and flipflopping
if, e.g., quotes are used in titles.
Closesjgm/citeproc#87.
We were having trouble building on ghc 8.4 because of
the lack of a Foldable instance for (Alt Maybe) in
base < 4.12.
Mystery: for some reason our builds were failing for gitit
but not in the pandoc CI.
We only depend on the urlEncode function in the package, which is also
provided by http-types. The HTTP package also depends on the network
package, which has difficulty building on ghcjs.
Add internal module Text.Pandoc.Network.HTTP, exporting `urlEncode`.
- Add manual entry for (non-default) extension
`rebase_relative_paths`.
- Add constructor `Ext_rebase_relative_paths` to `Extensions`
in Text.Pandoc.Extensions [API change]. When enabled, this
extension rewrites relative image and link paths by prepending
the (relative) directory of the containing file.
- Make Markdown reader sensitive to the new extension.
- Add tests for #3752.
Closes#3752.
NB. currently the extension applies to markdown and associated
readers but not commonmark/gfm.
* Allow spaces and most unicode characters in attachment links.
* No longer require a newline character after `{noformat}`.
* Only allow URI path segment characters in bare links.
* The `file:` schema is no longer allowed in bare links; these
rarely make sense.
Closes: #7218
Previously, when multiple file arguments were provided, pandoc
simply concatenated them and passed the contents to the readers,
which took a Text argument.
As a result, the readers had no way of knowing which file
was the source of any particular bit of text. This meant that
we couldn't report accurate source positions on errors or
include accurate source positions as attributes in the AST.
More seriously, it meant that we couldn't resolve resource
paths relative to the files containing them
(see e.g. #5501, #6632, #6384, #3752).
Add Text.Pandoc.Sources (exported module), with a `Sources` type
and a `ToSources` class. A `Sources` wraps a list of `(SourcePos,
Text)` pairs. [API change] A parsec `Stream` instance is provided for
`Sources`. The module also exports versions of parsec's `satisfy` and
other Char parsers that track source positions accurately from a
`Sources` stream (or any instance of the new `UpdateSourcePos` class).
Text.Pandoc.Parsing now exports these modified Char parsers instead of
the ones parsec provides. Modified parsers to use a `Sources` as stream
[API change].
The readers that previously took a `Text` argument have been
modified to take any instance of `ToSources`. So, they may still
be used with a `Text`, but they can also be used with a `Sources`
object.
In Text.Pandoc.Error, modified the constructor PandocParsecError
to take a `Sources` rather than a `Text` as first argument,
so parse error locations can be accurately reported.
T.P.Error: showPos, do not print "-" as source name.
* Build `+RTS -A256m -RTS` into default ghc-options for benchmark,
so we don't have to specify this separately on the command line.
This is necessary to get accurate benchmark results; otherwise
we are largely measuring garbage collecting, some not related
to the current benchmark.
* Switch back from gauge to tasty-bench.
* Allow specifying BASELINE file in 'make bench' for comparison
(otherwise the latest is chosen by default).
* Remove obsolete reference to weigh-pandoc from CONTRIBUTING.md.
* Remove `-Rghc-timing` from 'make bench'.
Jira reader:
* Fixed parsing of autolinks (i.e., of bare URLs in the text).
Previously an autolink would take up the rest of a line, as spaces
were allowed characters in these items.
* Emoji character sequences no longer cause parsing failures. This was
due to missing backtracking when emoji parsing fails.
Jira writer:
* Block quotes are only rendered as `bq.` if they do not contain a
linebreak.
This reverts commit 7a1d0f01e9.
This option gives confusing output when a build is interrupted,
suggesting that packages aren't required when we just didn't
get to the model that requires them.
This version of skylighting uses xml-conduit rather than hxt.
This speeds up parsing of XML syntax definitions fourfold, and
removes four packages from pandoc's dependency graph:
hxt-charproperties
hxt-unicode
hxt-regex-xmlschema
hxt
..and add new definitions isomorphic to xml-light's, but with
Text instead of String. This allows us to keep most of the code in
existing readers that use xml-light, but avoid lots of unnecessary
allocation.
We also add versions of the functions from xml-light's
Text.XML.Light.Output and Text.XML.Light.Proc that operate
on our modified XML types, and functions that convert
xml-light types to our types (since some of our dependencies,
like texmath, use xml-light).
Update golden tests for docx and pptx.
OOXML test: Use `showContent` instead of `ppContent` in `displayDiff`.
Docx: Do a manual traversal to unwrap sdt and smartTag.
This is faster, and needed to pass the tests.
Benchmarks:
A = prior to 8ca191604d (Feb 8)
B = as of 8ca191604d (Feb 8)
C = this commit
| Reader | A | B | C |
| ------- | ----- | ------ | ----- |
| docbook | 18 ms | 12 ms | 10 ms |
| opml | 65 ms | 62 ms | 35 ms |
| jats | 15 ms | 11 ms | 9 ms |
| docx | 72 ms | 69 ms | 44 ms |
| odt | 78 ms | 41 ms | 28 ms |
| epub | 64 ms | 61 ms | 56 ms |
| fb2 | 14 ms | 5 ms | 4 ms |
* Modified the Doc parser to skip leading blank lines. This fixes
parsing of documents which start with multiple blank lines.
(#7095)
* Prevent URLs within link aliases to be treated as autolinks.
(#6944)
Fixes: #7095Fixes: #6944
This exports functions that uses xml-conduit's parser to
produce an xml-light Element or [Content]. This allows
existing pandoc code to use a better parser without
much modification.
The new parser is used in all places where xml-light's
parser was previously used. Benchmarks show a significant
performance improvement in parsing XML-based formats
(especially ODT and FB2).
Note that the xml-light types use String, so the
conversion from xml-conduit types involves a lot
of extra allocation. It would be desirable to
avoid that in the future by gradually switching
to using xml-conduit directly. This can be done
module by module.
The new parser also reports errors, which we report
when possible.
A new constructor PandocXMLError has been added to
PandocError in T.P.Error [API change].
Closes#7091, which was the main stimulus.
These changes revealed the need for some changes
in the tests. The docbook-reader.docbook test
lacked definitions for the entities it used; these
have been added. And the docx golden tests have been
updated, because the new parser does not preserve
the order of attributes.
Add entity defs to docbook-reader.docbook.
Update golden tests for docx.
This change allows bibtex/biblatex output to wrap as other
formats do, depending on the settings of `--wrap` and `--columns`.
It also introduces default templates for bibtex and biblatex,
which allow for using the variables `header-include`, `include-before`
or `include-after` (or alternatively the command line options
`--include-in-header`, `--include-before-body`, `--include-after-body`)
to insert content into the generated bibtex/biblatex.
This change requires a change in the return type of the unexported
`T.P.Citeproc.writeBibTeXString` from `Text` to `Doc Text`.
Closes#7068.
* `biblatex` and `bibtex` are now supported as output
as well as input formats.
* New module Text.Pandoc.Writers.BibTeX, exporting
writeBibTeX and writeBibLaTeX. [API change]
* New unexported function `writeBibtexString` in
Text.Pandoc.Citeproc.BibTeX.
Allow defaults files to inherit options from other defaults files by
specifying them with the following syntax:
`defaults: [list of defaults files or single defaults file]`.
