We previously did this only with raw blocks, on the assumption
that math environments would always be raw blocks. This has changed
since we now parse them as inline environments.
Closes#3816.
Thus, a span with attribute 'foo' gets written to HTML5
with 'data-foo', so it is valid HTML5.
HTML4 is not affected.
This will allow us to use custom attributes in pandoc without
producing invalid HTML.
Fixed applyMacros so that it operates on the whole
string, not just the first token!
Don't remove macro definitions from the output,
even if Ext_latex_macros is set, so that macros will
be applied. Since they're only applied to math in
Markdown, removing the macros can have bad effects.
Even for math macros, keeping them should be harmless.
Added TikiWiki reader, including tests and documentation.
It's probably not *complete*, but it works pretty well, handles all
the basics (and some not-so-basics).
This rewrite is primarily motivated by the need to
get macros working properly. A side benefit is that the
reader is significantly faster (27s -> 19s in one
benchmark, and there is a lot of room for further
optimization).
We now tokenize the input text, then parse the token stream.
Macros modify the token stream, so they should now be effective
in any context, including math. Thus, we no longer need the clunky
macro processing capacities of texmath.
A custom state LaTeXState is used instead of ParserState.
This, plus the tokenization, will require some rewriting
of the exported functions rawLaTeXInline, inlineCommand,
rawLaTeXBlock.
* Added Text.Pandoc.Readers.LaTeX.Types (new exported module).
Exports Macro, Tok, TokType, Line, Column. [API change]
* Text.Pandoc.Parsing: adjusted type of `insertIncludedFile`
so it can be used with token parser.
* Removed old texmath macro stuff from Parsing.
Use Macro from Text.Pandoc.Readers.LaTeX.Types instead.
* Removed texmath macro material from Markdown reader.
* Changed types for Text.Pandoc.Readers.LaTeX's
rawLaTeXInline and rawLaTeXBlock. (Both now return a String,
and they are polymorphic in state.)
* Added orgMacros field to OrgState. [API change]
* Removed readerApplyMacros from ReaderOptions.
Now we just check the `latex_macros` reader extension.
* Allow `\newcommand\foo{blah}` without braces.
Fixes#1390.
Fixes#2118.
Fixes#3236.
Fixes#3779.
Fixes#934.
Fixes#982.
in Text.Pandoc.Lua. Also to pushPandocModule.
This change allows users to override pandoc.lua with a file
in their local data directory, adding custom functions, etc.
@tarleb, if you think this is a bad idea, you can revert this.
But in general our data files are all overridable.
No more SingleQuoted, DoubleQuoted, InlineMath, DisplayMath.
This makes everything uniform and predictable, though it does
open up a difference btw lua filters and custom writers.
If the metadata field is all on one line, we try to interpret
it as Inlines, and only try parsing as Blocks if that fails.
If it extends over one line (including possibly the `|` or
`>` character signaling an indented block), then we parse as
Blocks.
This was motivated by some German users finding that
date: '22. Juin 2017'
got parsed as an ordered list.
Closes#3755.
Note that if the table has a first page header and a
continuation page header, the notes will appear only
on the first occurrence of the header.
Closes#2378.
* New module Text.Pandoc.Readers.Vimwiki, exporting readVimwiki [API change].
* New input format `vimwiki`.
* New data file, `data/vimwiki.css`, for displaying the HTML produced by this reader and pandoc's HTML writer in the style of vimwiki's own HTML export.
Note that as a result of this change, the following,
which formerly produced a header with two lines separated
by a line break, will now produce a header followed by a
paragraph:
# Hi\
there
This may affect some existing documents that relied on
this undocumented and unintended behavior.
This change makes pandoc more consistent with other
Markdown implementations, and with itself (since the two-space
version of a line break doesn't work inside ATX headers, and
neither version works inside Setext headers).
Closes#3730.
* XML.toEntities: changed type to Text -> Text.
* Shared.tabFilter -- fixed so it strips out CRs as before.
* Modified writers to take Text.
* Updated tests, benchmarks, trypandoc.
[API change]
Closes#3731.
Currently we only handle the form `0.9\linewidth`.
Anything else would have to be converted to a percentage,
using some kind arbitrary assumptions about line widths.
See #3709.
The Emacs default is to include tags in the headline when exporting.
Instead of just empty spans, which contain the tag name as attribute,
tags are rendered as small caps and wrapped in those spans.
Non-breaking spaces serve as separators for multiple tags.
Babel result blocks can have block attributes like captions and names.
Result blocks with attributes were not recognized and were parsed as
normal blocks without attributes.
Fixes: #3706
Until now, org-ref cite keys included special characters also at the
end. This caused problems when citations occur right before colons or
at the end of a sentence.
With this change, all non alphanumeric characters at the end of a cite
key are ignored.
This also adds `,` to the list of special characters that are legal
in cite keys to better mirror the behaviour of org-export.
With `--reference-location` of `section` or `block`, pandoc
will now repeat references that have been used in earlier
sections.
The Markdown reader has also been modified, so that *exactly*
repeated references do not generate a warning, only
references with the same label but different targets.
The idea is that, with references after every block,
one might want to repeat references sometimes.
Closes#3701.
Emacs parses org documents into a tree structure, which is then
post-processed during exporting. The reader is changed to do the same,
turning the document into a single tree of headlines starting at
level 0.
Fixes: #3695
- Export `inEm` from ImageSize [API change].
- Change `showFl` and `show` instance for `Dimension` so
extra decimal places are omitted.
- Added `Em` as a constructor of `Dimension` [API change].
- Allow `em`, `cm`, `in` to pass through without conversion
in HTML, LaTeX.
Closes#3450.
This is now the default for pandoc's Markdown.
It allows whitespace between the two parts of a
reference link: e.g.
[a] [b]
[b]: url
This is now forbidden by default.
Closes#2602.
This is a verison of parseFromString specialied to
ParserState, which resets stateLastStrPos at the end.
This is almost always what we want.
This fixes a bug where `_hi_` wasn't treated as emphasis in
the following, because pandoc got confused about the
position of the last word:
- [o] _hi_
Closes#3690.
Parsing of smart quotes and special characters can either be enabled via
the `smart` language extension or the `'` and `-` export options. Smart
parsing is active if either the extension or export option is enabled.
Only smart parsing of special characters (like ellipses and en and em
dashes) is enabled by default, while smart quotes are disabled.
This means that all smart parsing features will be enabled by adding the
`smart` language extension. Fine-grained control is possible by leaving
the language extension disabled. In that case, smart parsing is
controlled via the aforementioned export OPTIONS only.
Previously, all smart parsing was disabled unless the language extension
was enabled.
Support for the `#+INCLUDE:` file inclusion mechanism was added.
Recognized include types are *example*, *export*, *src*, and normal org
file inclusion. Advanced features like line numbers and level selection
are not implemented yet.
Closes: #3510
The grid table parsers for markdown and rst was combined into one single
parser, slightly changing parsing behavior of both parsers:
- The markdown parser now compactifies block content cell-wise: pure
text blocks in cells are now treated as paragraphs only if the cell
contains multiple paragraphs, and as plain blocks otherwise. Before,
this was true only for single-column tables.
- The rst parser now accepts newlines and multiple blocks in header
cells.
Closes: #3638
* LaTeX: Load `parskip` before `hyperref`.
According to `hyperref` package's `README.pdf`, page 22, `hyperref` package
should be loaded after `parskip` package.
* Adjust tests for previous change.
Previously we inadvertently interpreted indented HTML as
code blocks. This was a regression.
We now seek to determine the indentation level of the contents
of an HTML block, and (optionally) skip that much indentation.
As a side effect, indentation may be stripped off of raw
HTML blocks, if `markdown_in_html_blocks` is used. This
is better than having things interpreted as indented code
blocks.
Closes#1841.
* Fix keyval funtion: pandoc did not parse options in braces correctly. Additionally, dot, dash, and colon were no valid characters
* Add | as possible option value
* Improved code
Previously the Markdown writer would sometimes create links where there
were none in the source. This is now avoided by selectively escaping bracket
characters when they occur in a place where a link might be created.
Closes#3619.
Ensure that we do not generate reference links
whose labels differ only by case.
Also allow implicit reference links when the link
text and label are identical up to case.
Closes#3615.
The implicitly defined global filter (i.e. all element filtering
functions defined in the global lua environment) is used if no filter is
returned from a lua script. This allows to just write top-level
functions in order to define a lua filter. E.g
function Emph(elem) return pandoc.Strong(elem.content) end
Previously the LaTeX writer created invalid LaTeX
when `--listings` was specified and a code span occured
inside emphasis or another construction.
This is because the characters `%{}\` must be escaped
in lstinline when the listinline occurs in another
command, otherwise they must not be escaped.
To deal with this, adoping Michael Kofler's suggestion,
we always wrap lstinline in a dummy command `\passthrough`,
now defined in the default template if `--listings` is
specified. This way we can consistently escape the
special characters.
Closes#1629.
A single `read` function parsing pandoc-supported formats is added to
the module. This is simpler and more convenient than the previous method
of exposing all reader functions individually.
We now issue `<div class="line-block">` and include a
default definition for `line-block` in the default
templates, instead of hard-coding a `style` on the
div.
Closes#1623.
A figure with two subfigures turns into two pandoc
figures; the subcaptions are used and the main caption
ignored, unless there are no subcaptions.
Closes#3577.
Source block parameter names are no longer prefixed with *rundoc*. This
was intended to simplify working with the rundoc project, a babel
runner. However, the rundoc project is unmaintained, and adding those
markers is not the reader's job anyway.
The original language that is specified for a source element is now
retained as the `data-org-language` attribute and only added if it
differs from the translated language.
The line-numbering switch that can be given to source blocks (`-n` with
an start number as an optional parameter) is parsed and translated to a
class/key-value combination used by highlighting and other readers and
writers.
Previously we always added an empty div before the list
item, but this created problems with spacing in tight
lists. Now we do this:
If the list item contents begin with a Plain block,
we modify the Plain block by adding a Span around
its contents.
Otherwise, we add a Div around the contents of the
list item (instead of adding an empty Div to the
beginning, as before).
Closes#3596.
Filtering functions take element components as arguments instead of the
whole block elements. This resembles the way elements are handled in
custom writers.
Instead of taking the whole inline element, forcing users to destructure it
themselves, the components of the elements are passed to the filtering
functions.
Plain text readers are exposed to lua scripts via the `pandoc.reader`
submodule, which is further subdivided by format. Converting e.g. a
markdown string into a pandoc document is possible from within lua:
doc = pandoc.reader.markdown.read_doc("Hello, World!")
A `read_block` convenience function is provided for all formats,
although it will still parse the whole string but return only the first
block as the result.
Custom reader options are not supported yet, default options are used
for all parsing operations.
Closes#3547.
Macro definitions are inserted in the template when there is highlighted
code.
Limitations: background colors and underline currently not
supported.
This avoids overly narrow labels for ordered lists with
() delimiters.
However, arguably it creates overly wide labels for bullets.
Also, lists now start flush with the margin, rather than
indented.
Fixes#2421.
Modified template to include a `<back>` and `<body>` section.
This should give authors more flexibility, e.g. to put
acknowledgements metadata in `<back>`. References are
automatically extracted and put into `<back>`.
* Fix lstinline handling: lstinline with braces can be used (verb cannot be used with braces)
* Use codeWith and determine the language from lstinline
* Improve code
* Add another test: convert lstinline without language option
* New module: Text.Pandoc.Writers.Ms.
* New template: default.ms.
* The writer uses texmath's new eqn writer to convert math
to eqn format, so a ms file produced with this writer
should be processed with `groff -ms -e` if it contains
math.
This is enabled by default in pandoc and GitHub markdown but not the
other flavors.
This requirse a space between the opening #'s and the header
text in ATX headers (as CommonMark does but many other implementations
do not). This is desirable to avoid falsely capturing things ilke
#hashtag
or
#5Closes#3512.
* Add `--lua-filter` option. This works like `--filter` but takes pathnames of special lua filters and uses the lua interpreter baked into pandoc, so that no external interpreter is needed. Note that lua filters are all applied after regular filters, regardless of their position on the command line.
* Add Text.Pandoc.Lua, exporting `runLuaFilter`. Add `pandoc.lua` to data files.
* Add private module Text.Pandoc.Lua.PandocModule to supply the default lua module.
* Add Tests.Lua to tests.
* Add data/pandoc.lua, the lua module pandoc imports when processing its lua filters.
* Document in MANUAL.txt.
This was failing because of a small discrepancy in markdown
table header line lengths on appveyor.
It's a minor issue, I can't see what is causing it, and
it's irrelevant to the issue this is testing, so we'll
just write native for this test.
Closes#1905.
Removed stateChapters from ParserState.
Now we parse chapters as level 0 headers, and parts as level -1 headers.
After parsing, we check for the lowest header level, and if it's
less than 1 we bump everything up so that 1 is the lowest header level.
So `\part` will always produce a header; no command-line options
are needed.
Previously only autogenerated ids were added to the list
of header identifiers in state, so explicit ids weren't taken
into account when generating unique identifiers. Duplicated
identifiers could result.
This simple fix ensures that explicitly given identifiers are
also taken into account.
Fixes#1745.
Note some limitations, however. An autogenerated identifier
may still coincide with an explicit identifier that is given
for a header later in the document, or with an identifier on
a div, span, link, or image. Fixing this would be much more
difficult, because we need to run `registerHeader` before
we have the complete parse tree (so we can't get a complete
list of identifiers from the document by walking the tree).
However, it might be worth issuing warnings for duplicate
header identifiers; I think we can do that. It is not
common for headers to have the same text, and the issue
can always be worked around by adding explicit identifiers,
if the user is aware of it.
* Previously we got overlong lists with `--wrap=none`. This is fixed.
* Previously a multiline list could become a simple list (and would
always become one with `--wrap=none`).
Closes#3384.
when they occur without space surrounding them.
E.g. equation, math.
This avoids incorrect vertical space around equations.
Closes#3309.
Closes#2171.
See also rstudio/bookdown#358.
E.g. an HTML table with two cells in the first row and one
in the second (but no row/colspan).
We now calculate the number of columns based on the longest
row (or the length of aligns or widths).
Closes#3337.
When multiple YAML metadata blocks are used, and two define
the same field, the value defined first takes precedence,
according to the manual. This was changed briefly in
ba3ee62323. This commit
reverts to the original behavior and adds a test case.
All templates now include `code{white-space: pre-wrap}`
and CSS for `q` if `--html-q-tags` is used.
Previously some templates had `pre` and others `pre-wrap`;
the `q` styles were only sometimes included.
See #3485.
Added test cases.
Fixed HTML reader to parse a span with class "smallcaps" as
SmallCaps.
Fixed Markdown writer to render SmallCaps as a native span
when native spans are enabled.
Polyglot markup is HTML5 that is also valid XHTML. See
<https://www.w3.org/TR/html-polyglot>. With this change, pandoc's
html5 writer creates HTML that is both valid HTML5 and valid XHTML.
See jgm/pandoc-templates#237 for prior discussion.
* Add xml namespace to `<html>` element.
* Make all `<meta>` elements self closing.
See <https://www.w3.org/TR/html-polyglot/#empty-elements>.
* Add `xml:lang` attribute on `<html>` element, defaulting to blank, and
always include `lang` attribute, even when blank. See
<https://www.w3.org/TR/html-polyglot/#language-attributes>.
* Update test files for template changes.
The key justification for having language values default to blank: it
turns out the HTML5 spec requires it (as I read it). Under
[the HTML5 spec, section "3.2.5.3. The lang and xml:lang
attributes"](https://www.w3.org/TR/html/dom.html#the-lang-and-xmllang-attributes),
providing attributes with blank contents both:
* Has meaning, "unknown", and
* Is a MUST (written as "must") if a language value is not provided ...
> The lang attribute (in no namespace) specifies the primary language
> for the element's contents and for any of the element's attributes that
> contain text. Its value must be a valid BCP 47 language tag, or the
> empty string. Setting the attribute to the empty string indicates that
> the primary language is unknown.
In short, it seems that where a language value is not provided then a
blank value MUST be provided for Polyglot Markup conformance, because
the HTML5 spec stipulates a "must". So although the Polyglot Markup spec
is unclear on this issue it would seem that if it was correctly written,
it would therefore require blank attributes.
Further justifications are found at
https://github.com/jgm/pandoc-templates/issues/237#issuecomment-275584181
(but the HTML5 spec justification given above would seem to be the
clincher).
In addition to having lang-values-default-to-blank I recommend that, when an
author does not provide a lang value, then upon on pandoc command execution
a warning message like the following be provided:
> Polyglot markup stipulates that 'The root element SHOULD always specify
> the language'. It is therefore recommended you specify a language value in
> your source document. See
> <https://www.w3.org/International/articles/language-tags/> for valid
> language values.
The citations appear at the end of the document as a definition
list in a special div with id `citations`.
Citations link to the definitions.
Added stateCitations to ParserState.
Closes#853.
Previously we would refuse to parse anything as raw inline if
it was in the blockCommands list. Now we allow exceptions
if they're listed under ignoreInlines in inlineCommands.
This should make it easier e.g. to include an \hspace
between two side-by-side raw LaTeX tables.
Otherwise things like `\noindent foo` break and turn into
`\noindentfoo`.
Affects `-f latex+raw_tex` and `-f markdown` (and other formats
that allow `raw_tex`).
Closes#1773.
For example, in
\begin{tabular}{>{$}l<{$}>{$}l<{$} >{$}l<{$}}
each cell will be interpreted as if it has a `$`
before its content and a `$` after (math mode).
Currently the support for the `.. table` directive is a bit
limited; we don't yet support the `widths` field. But at least
you can have a proper captioned table.
These were confusing.
Now we rely on the +raw_tex or +raw_html extension with latex
or html input.
Thus, instead of
--parse-raw -f latex
we use
-f latex+raw_tex
and instead of
--parse-raw -f html
we use
-f html+raw_html
The org writer was inserting two spaces after list bullets. Emacs
Org-mode defaults to a single space, so behavior is changed to reflect
this.
Closes: #3417
