New input format: endnotexml
New reader module: Text.Pandoc.Readers.EndNote, exporting
`readEndNoteXML` and `readEndNoteXMLReferences`. [API change]
This reader is still a bit rudimentary, but it should get
be good enough to be helpful.
These are supported in the same way as Zotero citations,
using the same code. As with Zotero, enable the `citations`
extension on `docx` to parse these as native citations.
Closes#7840.
Previously we used System.FilePath's isRelative to
determine when paths are relative (since absolute
paths need to get a new name based on the sha1 hash).
But this has an OS-specific behavior and actually
returns True on Windows for paths like `/media/file.png`.
This ought to fix#7881.
When enabled, Zotero citations are parsed as native pandoc
citations. (When disabled, the Zotero-generated citation
text is passed through as regular text.) In addition, the
Zotero-generated bibliography is suppressed.
Locators still need some work.
- Add docxReferences to state, so we can accumulate
references for metadata.
- Add a clause for ZoteroItem to parPartToInlines'.
So far it doesn't do anything except add a surrounding Cite element.
See #7840.
This reverts commit 545c0911aa.
Fixes regression in 2.17.1.
The original commit was completely misguided, and caused
problems finding defaults files in the default user data
directory.
Attribute key-value pairs are marshaled as AttributeList, i.e., as a
userdata type that behaves both like a list and a map. This allows to
preserve the order of key-value pairs.
Closes: #7489
If a file path does not exist relative to the working directory, but
it does exist relative to the user data directory, and it exists outside
of the user data directory, do not read it. This applies to readDataFile
and readMetadataFile in PandocMonad and, by extension, any module that
uses these by passing them relative paths.
Adding a pair of braces around the second argument of `\def`
prevents LaTeX from an emergency stop with: Closes#6096.
```
pandoc -f markdown -o test.pdf
\newif\ifepub
\epubtrue
\ifepub
hi
\fi
^D
```
If files specified with `--metadata-file` are not found in the working
directory, look in `$DATADIR/metadata`.
Expose new `readMetadataFile` function from Text.Pandoc.Class
[API change].
Expose new `PandocCouldNotFindMetadataFileError` constructor for
`PandocError` from Text.Pandoc.Error [API change].
Closes#5876.
This was causing serious problems with `newif` commands.
See #6096. And it didn't seem to make any difference for
the tests; I assume that, unless there's some untested
behavior, this is something that has now become unnecessary.
If speaker notes (a Div with class 'notes') occur right
after a section heading, but above slide level, the
resulting `\note{..}` caommand should not be wrapped in
a frame, as that will cause a spurious blank slide.
Closes#7857.
With the new (default) line wrapping of HTML, in
conjunction with the default CSS which includes
`code { whitespace: pre-wrap; }`, spurious line
breaks could be introduced into inline code.
Closes#7858.
Otherwise we'll sometimes get two copies of things, one
from the `citationPrefix` or `citationSuffix` and another
from the embedded fallback text.
When there is no fallback text, we'll get no content.
However, it really isn't an alternative to just rely
on the result of running `query` on the embedded `Citation`s;
this will result in a jumble of text rather than anything
structured.
Closes#7855.
So far this just adds a constructor for FieldInfo;
we'll need to adjust the rest of the reader code to
parse the JSON and do something with it.
See #7840.
If a table has explicit column width information *and* the
content extends beyond the `--columns` width, we need to
adjust the widths of the pipe separators to encode this width
information.
Closes#7847.
Adjacent docx tables need to be separated by an empty paragraph. If
there's a RawBlock between tables which renders to nothing, be sure to
still insert the empty paragraph so that they will not collapse
together.
Fixes#7724
Previously we would (also) search the default user data directory
for a defaults file, even if a different user data directory
was specified using `--data-dir`. This was a mistake; if
`--data-dir` is used, the default user data directory should
not be searched.
If the XDG data directory is not defined (e.g. because
it's not supported in the OS or HOME isn't defined), we
return the empty string instead of raising an exception.
Closes#7842.
The V font is defined conditionally, so that it renders
like CB in output formats that support that, and like B
in those that don't (e.g. the terminal).
We could just redefine C, but this would affect code
blocks, too, and putting them all in boldface looks ugly,
I think.
Possible drawback: fragments created by pandoc's man
writer will presuppose a nonstandard V font.
Closes#7506.
Supersedes 253467a549.
Closes#7506.
This also allows us to get rid of some special casing
on definition lists that ensured that options in code
spans would be boldface. (If this change is ever reverted,
we'll need that again.)
with numerical styles that use superscripts (e.g.
american-medical-association.csl), as well as with
note styles. The default setting of `notes-after-punctuation`
is true for note styles and false otherwise.
This restores a behavior of pandoc-citeproc that wasn't properly
carried over to Citeproc.
Closes#7826.
See also jgm/pandoc-citeproc#384.
Make sure that we only create one bullet per list item in docx. In
particular, when a div is a list item, its contained paragraphs will
now no longer wrongly get individual bullets.
This is accomplished by making sure that for each list, we only use
the associated numId once. Any repeated use would add incorrect
bullets to the document.
Closes#7689
The commit 7a9832166e
had the effect that blank lines would be collapsed
in HTML attributes.
We also roll back a change that collapsed multiple
spaces into one.
* Add localized title "abstract", unless `abstract-title` variable
is set.
* Add `abstract-title` div to abstract CSS.
* Move abstract CSS out of CSL conditional.
* Ensure that abstract is aligned left but indented on all sides.
* Use smaller font for abstract.
Improves #7588.
This adds support for alphabetical lists in org by enabling the
extension Ext_fancy_lists, mimicking the behaviour of Org Mode when
org-list-allow-alphabetical is enabled.
Enabling Ext_fancy_lists will also make Pandoc differentiate between the
delimiters of ordered lists (periods or closing parentheses). Org does
this differentiation by default when exporting to some formats (e.g.
plain text) but does not in others (e.g. html and latex), so I decided
to copy Pandoc's markdown reader behaviour.
Prior to this commit the MediaWiki writer always added the display
text for a wiki link:
* [[Help|Help]]
* [[Bubbles|Everyone loves bubbles]]
However the display text in the first example is redundant since
MediaWiki uses the target as the default display text. The result being:
* [[Help]]
* [[Bubbles|Everyone loves bubbles]]
This adds support for counter cookies in org lists. Such cookies are
used to override the item counter in ordered lists. In org it is
possible to set the counter at any list item, but since Pandoc AST does
not support this, we restrict the usage to setting an offset for the
entire ordered list, by using the cookie in the first list item.
Note that even though unordered lists do not have counters, Org Mode
still parses such cookies in unordered lists and suppresses them in the
output, so we do the same.
Also, even though org-list-allow-alphabetical is disabled in Emacs by
default, for some reason alphabetical cookies are always parsed and used
in Org Mode regardlessly of whether this option is enabled or the list
style is decimal, so we do the same.
E.g.
2. test
3. test
Is parsed as an ordered list starting at 1, as before. This also
conforms to Org Mode behaviour.
1. [@2] test
2. test
Is now parsed as an ordered list starting at 2, so that it conforms to
Org Mode behaviour.
Note that when parsing
1. [@2] test
2. [@9] test
the second cookie is silenced and the entire list starts at 2. This is
because the current Pandoc AST does not support expressing a change in
the counter at a specific item.
This ensures that all writer exported in T.P.Writers are parameterized
and work with any `PandocMonad` type. This is consistent with
T.P.Readers, as `readCustom` is not exported from that module either.
New templates variables can be added by giving variable-value pairs as a
second return value of the global function `Doc`.
Example:
function Doc (body, meta, vars)
vars.date = vars.date or os.date '%B %e, %Y'
return body, vars
end
Closes: #6731
API changes:
- The function T.P.Filter.applyFilters now takes a filter
environment of type `Environment`, instead of a ReaderOptions value.
The `Environment` type is exported from `T.P.Filter` and allows to
combine ReaderOptions and WriterOptions in a single value.
- Global, exported from T.P.Lua, has a new type constructor
`PANDOC_WRITER_OPTIONS`.
Closes: #5221
The function `pandoc.read` is updated to use the same state that was
used while parsing the main input files. This ensures that log messages
are preserved and that images embedded in the input are added to the
mediabag.
...instead of ParPart.
Also remove NullParPart constructor, as it is no longer
needed.
This will allow us to handle elements that contain multiple
ParParts, e.g. w:drawing elements with multiple pic:pic.
See #7786.
We were using `splitPath` in two places in the code
where `splitDirectories` should have been used.
This led to a test for `..` in paths in `extractMedia`
failing, so that images with `..` in the path name
could be extracted outside the directory specified
by `extractMedia`.
It also led a test for `media` in resource paths to fail
in the docx reader.
Previously some displayed formulas would be floated above
a preceding text line. This is fixed by setting vertical-rel
to 'text' rather than 'paragraph-content'.
Closes#7777.
- With `--wrap=none`, we now output line breaks between
block-level elements. Previously they were omitted
entirely, so the whole document was on one line, unless
there were literal line breaks in pre sections. This makes
the HTML writer's behavior more consistent with that of
other writers.
- Put newline after `<dd>`.
- Put newlines after block-level elements in footnote section.
Previously the HTML writer was exceptional in not being
sensitive to the `--wrap` option. With this change `--wrap`
now works for HTML. The default (as with other formats) is
automatic wrapping to 72 columns.
A new internal module, T.P.Writers.Blaze, exports `layoutMarkup`.
This converts a blaze Html structure into a doclayout Doc Text.
In addition, we now add a line break between an `img` tag
and the associated `figcaption`.
Note: Output is never wrapped in `writeHtmlStringForEPUB`.
This accords with previous behavior since previously the HTML
writer was insensitive to `--wrap` settings. There's no real
need to wrap HTML inside a zipped container.
Note that the contents of script, textarea, and pre tags are
always laid out with the `flush` combinator, so that unwanted
spaces won't be introduced if these occur in an indented context
in a template.
Closes#7764.
The function behaves like the default `type` function from Lua's
standard library, but is aware of pandoc userdata types. A typical
use-case would be to determine the type of a metadata value.
The default Pandoc object is now non-strict, i.e., only the parts of the
document that are accessed will be marshaled to Lua. A special type is
no longer necessary.
This change also makes it possible to use the global variable with
library functions such as `pandoc.utils.references`, or to inspect the
document contents with `walk()`.
Markua is a markdown variant used by Leanpub.
More information about Markua can be found at https://leanpub.com/markua/read.
Adds a new exported function `writeMarkua` from T.P.Writers.Markdown.
[API change]
Closes#1871.
Co-authored by Tim Wisotzki and Samuel Lemmenmeier.
The JATS writer was losing quotes in element-citations, as it uses the
`T.P.Citeproc.getReferences` function to get references. That function
replaces `Quoted` elements with spans. That transformation is required
in `T.P.Citeproc.processCitations`, so it has been moved there.
...after author-in-text citations.
Previously `@item [p. 12; @item2]` was incorrectly parsed as
three citations rather than two. This is now fixed by ensuring
that `prefix` doesn't gobble any semicolons.
T.P.Citeproc exports `getCiteprocLang` and `getStyle` [API change].
T.P.Citeproc.Locator now exports `toLocatorMap`, `LocatorInfo`,
and `LocatorMap`. The type of `parseLocator` has changed, so
it now takes a `LocatorMap` rather than a `Locale` as parameter,
and returns a `LocatorInfo` instead of a tuple.
Previously, both `fmt == f` case and Image have a rank of 1.
In the end, e.g. from ipynb to html conversion,
if both html and image exists, it actually prefers the image.
This commit changes this, so that fmt == f is always highest rank,
and rank never collides.
This is achieved by keeping fmt == f case having rank 1,
and every other rank increased by 1.
Retry conversion by passing a string instead of sources when the
`Reader` fails with a message that hints at an outdated function. A
deprecation notice is reported in that case.
The first argument passed to Lua `Reader` functions is no longer a plain
string but a richer data structure. The structure can easily be
converted to a string by applying `tostring`, but is also a list with
elements that contain each the *text* and *name* of each input source as
a property of the respective name.
A small example is added to the custom reader documentation, showcasing
its use in a reader that creates a syntax-highlighted code block for
each source code file passed as input.
Existing readers must be updated.
Write RawBlock of markdown in code-cell output.
#7561 makes the ipynb reader reads code-cell output with mime
"text/markdown" to a RawBlock of markdown
This commit makes the ipynb writer writes this RawBlock of markdown
back inside a code-cell output with the same mime, preserving this
information in round-trip
Add tests of ipynb reader (#7561) and ipynb writer (#7563)'s ability to
handle a "text/markdown" mime type in a code-cell output
- `walk` methods are added to `Block` and `Inline` values; the methods
are similar to `pandoc.utils.walk_block` and
`pandoc.utils.walk_inline`, but apply to filter also to the element
itself, and therefore return a list of element instead of a single
element.
- Functions of name `Doc` are no longer accepted as alternatives for
`Pandoc` filter functions. This functionality was undocumented.
The new `pandoc.Inlines` function behaves identical on string input, but
allows other Inlines-like arguments as well.
The `pandoc.utils.text` function could be written as
function pandoc.utils.text (x)
assert(type(x) == 'string')
return pandoc.Inlines(x)
end
The marshaling functions for pandoc's AST are extracted into a separate
package. The package comes with a number of changes:
- Pandoc's List module was rewritten in C, thereby improving error
messages.
- Lists of `Block` and `Inline` elements are marshaled using the new
list types `Blocks` and `Inlines`, respectively. These types
currently behave identical to the generic List type, but give better
error messages. This also opens up the possibility of adding
element-specific methods to these lists in the future.
- Elements of type `MetaValue` are no longer pushed as values which
have `.t` and `.tag` properties. This was already true for
`MetaString` and `MetaBool` values, which are still marshaled as Lua
strings and booleans, respectively. Affected values:
+ `MetaBlocks` values are marshaled as a `Blocks` list;
+ `MetaInlines` values are marshaled as a `Inlines` list;
+ `MetaList` values are marshaled as a generic pandoc `List`s.
+ `MetaMap` values are marshaled as plain tables and no longer
given any metatable.
- The test suite for marshaled objects and their constructors has
been extended and improved.
- A bug in Citation objects, where setting a citation's suffix
modified it's prefix, has been fixed.
We were including the ams environment type in addition
to the number. This is proper behavior for `\cref` but
not for `\ref`. To support `\cref` we need to store
the environment label separately.
- Resolve references to theorem environments.
- Remove Span caused by "label" in figure, table, and theorem
environments; this had an id that duplicated the environments' id.
See #813.
Fixed calculation of maximum column widths in pipe tables.
It is now based on the length of the markdown line, rather
than a "stringified" version of the parsed line. This should
be more predictable for users. In addition, we take into account
double-wide characters such as emojis.
Closes#7713.
The function converts a string to `Inlines`, treating interword spaces
as `Space`s or `SoftBreak`s. If you want a `Str` with literal spaces,
use `pandoc.Str`.
Closes: #7709
Using a Lua string where a list of inlines is expected will cause the
string to be split into words, replacing spaces and tabs into
`pandoc.Space()` elements and newlines into `pandoc.SoftBreak()`.
The previous behavior was to treat the string `s` as `{pandoc.Str(s)}`.
The old behavior can be recovered by wrapping the string into a table
`{s}`.
Some people use `---` as the end delimiter in YAML
bibliography files, which causes the `yaml` library
to emit an error unless we explicitly allow multiple
YAML documents (and just consider the first).
In T.P.Readers.Metadata
We need to generate a span when the header's ID doesn't match
the one MediaWiki would generate automatically. But MediaWiki's
generation scheme is different from ours (it uses uppercase letters,
and `_` instead of `-`, for example).
This means that in going from markdown -> mediawiki, we'll now get
spans before almost every heading, unless explicit identifiers are
used that correspond to the ones MediaWiki auto-generates.
This is uglier output but it's necessary for internal links to
work properly.
See #7697.
The `lpeg` and `re` modules are loaded into globals of the respective
name, but they are not necessarily registered as loaded packages. This
ensures that
- the built-in library versions are preferred when setting the globals,
- a shared library is used if pandoc has been compiled without `lpeg`,
and
- the `require` mechanism can be used to load the shared library if
available, falling back to the internal version if possible and
necessary.
Ignore errors if the normal package mechanism failed; this not only
covers the case of modules being unavailable on the system, but also
works if the modules are present, but fail to load for some reason.
This makes the built-in package version a true fallback.
This allows us to get rid of the old custom prelude and
some crufty cpp. But the primary reason for this is that
conduit has bumped its base lower bound to 4.12, making it
impossible for us to support lower base versions.
Reader options can now be passed as an optional third argument to
`pandoc.read`. The object can either be a table or a ReaderOptions value
like `PANDOC_READER_OPTIONS`. Creating new ReaderOptions objects is
possible through the new constructor `pandoc.ReaderOptions`.
Closes: #7656
New module Text.Pandoc.Readers.Custom, exporting
readCustom [API change].
Users can now do `-f myreader.lua` and pandoc will treat the
script myreader.lua as a custom reader, which parses an input
string to a pandoc AST, using the pandoc module defined for
Lua filters.
A sample custom reader can be found in data/reader.lua.
Closes#7669.
* Support for <indexterm>s when reading DocBook
* Update implementation status of `<n-ary>` tags
* Remove non-idiomatic parentheses
* More complete `<indexterm>` support, with tests
Co-authored-by: Rowan Rodrik van der Molen <rowan@ytec.nl>
Compiles the 'lpeg' library (Parsing Expression Grammars For Lua) into
the program.
Package maintainers may choose to rely on package dependencies to make
lpeg available, in which case they can compile the with the constraint
`lpeg +rely-on-shared-lpeg-library`.
This is just a small improvement in terms of performance,
but it's simpler and more direct code.
Also, we avoid parsing interparagraph spaces in balanced
brackets, as the original did.
This ensures that when `SOURCE_DATE_EPOCH` is set, the
modification times of files taken from the reference.docx will
be set deterministically, allowing for reproducible builds.
Closes#7654.
Comparisons of Citation values are performed in Haskell; values are
equal if they represent the same Haskell value. Converting a Citation
value to a string now yields its native Haskell string representation.
E.g. "Y", "yes", which are now (with yaml library) considered
boolean values, as well as "null".
This fixes a bug with roundtripping markdown -> markdown:
```
---
foo: "true"
...
```
- For LineEnding use lowercase constructors, e.g. `crlf`, `native`.
This was the original intent, but there was a bug in the
implementation.
- For HTMLSlideVariant use lowercase constructors.
- For ReaderOptions use e.g. `default-image-extension`
instead of `readerDefaultImageExtension` for field names.
- For Extension, use e.g. `tex_math_dollars` instead of
`Ext_tex_math_dollars` as constructor.
- For Extensions, use an array of Extensions, instead of
an object wrapping the tag `Extensions` and an integer.
(The representation is not supposed to be part of the
public API.)
- For Opt, use field names like `tab-stop` instead of `optTabStop`.
Reasons:
- Performance: HsYAML is around 20 times slower in parsing
large YAML bibliographies (#6084).
- An issue was submitted to HsYAML, but it hasn't gotten
any attention. HsYAML seems borderline unmaintained; it hasn't
had a commit in over a year.
- Unfortunately this goes back on our attempts to free ourselves
from C dependencies (#4535). But I don't see a better alternative
until a better pure Haskell parser is available.
Closes#6084.
Notes:
- We've removed the FromYAML instances for all types that had
them, since this is a HsYAML-specific typeclass [API change].
(The yaml package just uses From/ToJSON.)
- Unlike HsYAML (in the configuration we were using), yaml
parses 'Y', 'N', 'Yes', 'No', 'On', 'Off' as boolean values.
Users may need to quote these when they are meant to be
interpreted as strings. Similarly, 'null' is parsed as
a YAML null value (and will be treated as an empty string
by pandoc rather than the string 'null'). Quoting it will
force it to be interpreted as a string.
- Some tests had to be adjusted accordingly.
- Pandoc now behaves better when the YAML metadata contains
escaping errors: instead of just falling back on treating
the section as a table, it raises a YAML parsing error.
Properties of Block values are marshalled lazily, which generally
improves performance considerably. Script users may also notice the
following differences:
- Block element properties can no longer be accessed by numerical
indexing of the `.c` field. The `.c` property now serves as an alias
for `.content`, so some filter that used this undocumented method
for property access may continue to work, while others will need to
be updated and use proper property names.
- The marshalled Block elements now have a `show` method, and a
`__tostring` metamethod. Both return the Haskell string
representation of the element.
- Block values now have the Lua type `userdata` instead of `table`.
This includes the following user-facing changes:
- Deprecated inline constructors are removed. These are `DoubleQuoted`,
`SingleQuoted`, `DisplayMath`, and `InlineMath`.
- Attr values are no longer normalized when assigned to an Inline
element property.
- It's no longer possible to access parts of Inline elements via
numerical indexes. E.g., `pandoc.Span('test')[2]` used to give
`pandoc.Str 'test'`, but yields `nil` now. This was undocumented
behavior not intended to be used in user scripts. Use named properties
instead.
- Accessing `.c` to get a JSON-like tuple of all components no longer
works. This was undocumented behavior.
- Only known properties can be set on an element value. Trying to set a
different property will now raise an error.
- Adds a new `pandoc.AttributeList()` constructor, which creates the
associative attribute list that is used as the third component of
`Attr` values. Values of this type can often be passed to constructors
instead of `Attr` values.
- `AttributeList` values can no longer be indexed numerically.
The new HsLua version takes a somewhat different approach to marshalling
and unmarshalling, relying less on typeclasses and more on specialized
types. This allows for better performance and improved error messages.
Furthermore, new abstractions allow to document the code and exposed
functions.
All the code we needed to put most styles and scripts into
inline style and script tags was there, but because of the
order of pattern matching, it was never being called.
Putting the catch-all clause at the end fixes the bug.
Closes#7635, closes#7367. See also #3423.
Previously pandoc would parse
[link to (@a)](url)
as a citation; similarly
[(@a)]{#ident}
This is undesirable. One should be able to use example references
in citations, and even if `@a` is not defined as an example
reference, `[@a](url)` should be a link containing an author-in-text
citation rather than a normal citation followed by literal `(url)`.
Closes#7632.
Some fields only have an instrText and no content, Pandoc didn't
understand these, causing other fields to be misunderstood because it
seemed like a field was still open when it wasn't.
Fields delimited by fldChar elements can contain other fields. Before,
the nested fields would be ignored, except for the end, which would be
considered the end of the parent field.
To fix this issue, fields needed to be considered containing ParParts
instead of Runs, since a Run can't represent complex enough structures.
This also impacted Hyperlinks since they can originate from a field.
This commit changes the `marL` and `indent` values used for plain
paragraphs and numbered lists, and changes the spacing defined in the
reference doc master for bulleted lists.
For paragraphs, there is now a left-indent taken from the `otherStyle`
in the master. For numbered lists, the number is positioned where the
text would be if this were a plain paragraph, and the text is indented
to the next level. This means that continuation paragraphs line up
nicely with numbered lists.
It also /mostly/ matches the observed PowerPoint behaviour when
inserting paragraphs and numbered lists: the only difference is that
PowerPoint was using a different margin value for the first level
numbered lists – I’ve changed this to match the other levels, as I don’t
think it makes the spacing unappealing and it allows continuation
paragraphs at any level to line up.
With bulleted lists, I’m keeping the observed PowerPoint behaviour of
specifying only a level, letting `marL` and `indent` be automatically
taken from `bodyStyle`. To that end, this commit changes the `bodyStyle`
spacing in the master of the default reference doc, to:
- line up the text of the first paragraph in each bullet with any
continuation paragraphs
- line up nested bullet markers in any continuation paragraphs with the
first paragraph, matching lists and plain paragraphs
This does mean the continuation paragraphs still won’t line up for
anyone using their own reference doc where they haven’t matched the
`otherStyle` and `bodyStyle` indent levels, but I think people in that
situation will be able to troubleshoot.
In PowerPoint, the content of a top-level list is at the same level as
the content of a top-level paragraph – the only difference is that a
list style has been applied.
At the moment, the pptx writer increments the paragraph level on each
list, turning what should be top-level lists into second-level lists.
This commit changes that logic, only incrementing the paragraph level on
continuation paragraphs of lists.
- Fixes https://github.com/jgm/pandoc/issues/4828
- Fixes https://github.com/jgm/pandoc/issues/4663
AsciiDoctor allows to request line numbering on code blocks by
using a switch on the `source` block, such as in:
```
[source%linesnum,haskell]
----
some Haskell code here
----
```
We used to attempt automatic sentence splitting in man and ms
output, since sentence-ending periods need to be followed by
two spaces or a newline in these formats.
But it's difficult to do this reliably at the level of
`[Inline]`.
When a paragraph has an indentation different from the parent (named)
style, it used to be considered a blockquote. But this only makes sense
when the paragraph has more indentation. So this commit adds a check
for the indentation of the parent style.
