We bypass the commonmark writer from cmark and construct our
own pipe tables, with better results. (Note also that cmark-gfm
currently doesn't support rendering table nodes; see
kivikakk/cmark-gfm-hs#3.)
when enabled (as with gfm). Note: because of limitations in
cmark-gfm, which will hopefully soon be corrected, this currently
gives an error on Tables.
Also properly support `--wrap=none`.
We no longer have a separate readGFM and writeGFM;
instead, we'll use readCommonMark and writeCommonMark
with githubExtensions.
It remains to implement these extensions conditionally.
Closes#3841.
This uses bindings to GitHub's fork of cmark, so it should parse
gfm exactly as GitHub does (excepting certain postprocessing
steps, involving notifications, emojis, etc.).
* Added Text.Pandoc.Readers.GFM (exporting readGFM)
* Added Text.Pandoc.Writers.GFM (exporting writeGFM)
* Added `gfm` as input and output forma
Note that tables are currently always rendered as HTML
in the writer; this can be improved when CMarkGFM supports
tables in output.
Also, fix regular macros so they're expanded at the
point of use, and NOT also the point of definition.
`\let` macros, by contrast, are expanded at the
point of definition. Added an `ExpansionPoint`
field to `Macro` to track this difference.
We used to parse paragraphs styled with "HeadingN" as "nth-level
header." But if a document has a custom style named "Heading0", this
will produce a 0-level header, which shouldn't exist. We only parse
this style if N>0. Otherwise we treat it as a normal style name, and
follow its dependencies, if any.
Closes#3830.
We previously did this only with raw blocks, on the assumption
that math environments would always be raw blocks. This has changed
since we now parse them as inline environments.
Closes#3816.
Thus, a span with attribute 'foo' gets written to HTML5
with 'data-foo', so it is valid HTML5.
HTML4 is not affected.
This will allow us to use custom attributes in pandoc without
producing invalid HTML.
It is no longer necessary, since the rawLaTeXBlock parser
will parse macro definitions.
This also avoids the need for a separate latexMacro parser
in the Markdown reader.
Fixed applyMacros so that it operates on the whole
string, not just the first token!
Don't remove macro definitions from the output,
even if Ext_latex_macros is set, so that macros will
be applied. Since they're only applied to math in
Markdown, removing the macros can have bad effects.
Even for math macros, keeping them should be harmless.
An unknown command at the beginning of the line that could
be either block or inline is treated as block if we have
a sequence of block commands followed by a newline or a
`\startXXX` command (which might start a raw ConTeXt environment).
Added TikiWiki reader, including tests and documentation.
It's probably not *complete*, but it works pretty well, handles all
the basics (and some not-so-basics).
We may want to think of some kind of graceful fallback, but
the present behavior has the advantage of forcing people to
update scripts when updating to pandoc 2.0.
See #3786.
This rewrite is primarily motivated by the need to
get macros working properly. A side benefit is that the
reader is significantly faster (27s -> 19s in one
benchmark, and there is a lot of room for further
optimization).
We now tokenize the input text, then parse the token stream.
Macros modify the token stream, so they should now be effective
in any context, including math. Thus, we no longer need the clunky
macro processing capacities of texmath.
A custom state LaTeXState is used instead of ParserState.
This, plus the tokenization, will require some rewriting
of the exported functions rawLaTeXInline, inlineCommand,
rawLaTeXBlock.
* Added Text.Pandoc.Readers.LaTeX.Types (new exported module).
Exports Macro, Tok, TokType, Line, Column. [API change]
* Text.Pandoc.Parsing: adjusted type of `insertIncludedFile`
so it can be used with token parser.
* Removed old texmath macro stuff from Parsing.
Use Macro from Text.Pandoc.Readers.LaTeX.Types instead.
* Removed texmath macro material from Markdown reader.
* Changed types for Text.Pandoc.Readers.LaTeX's
rawLaTeXInline and rawLaTeXBlock. (Both now return a String,
and they are polymorphic in state.)
* Added orgMacros field to OrgState. [API change]
* Removed readerApplyMacros from ReaderOptions.
Now we just check the `latex_macros` reader extension.
* Allow `\newcommand\foo{blah}` without braces.
Fixes#1390.
Fixes#2118.
Fixes#3236.
Fixes#3779.
Fixes#934.
Fixes#982.
You can now have the following fields in your YAML metadata,
and it will be treated appropriately in the generated
EPUB.
```
ibooks:
version: 1.3.4
specified-fonts: false
ipad-orientation-lock: portrait-only
iphone-orientation-lock: landscape-only
binding: true
scroll-axis: vertical
```
This commit also fixes a regression in stylesheet paths.
GitHub has two Markdown modes, one for long-form documents like READMEs
and one for short things like issue coments. In issue comments, a line
break is treated as a hard line break. In README, wikis, etc., it is
treated as a space as in regular Markdown.
Since pandoc is more likely to be used to convert long-form
documents from GitHub Markdown, `-hard_line_breaks` is a better
default.
Closes#3594.
in Text.Pandoc.Lua. Also to pushPandocModule.
This change allows users to override pandoc.lua with a file
in their local data directory, adding custom functions, etc.
@tarleb, if you think this is a bad idea, you can revert this.
But in general our data files are all overridable.
Also, now we check before running walkM that the function
table actually does contain something relevant. E.g. if
your filter just defines Str, there's no need to run walkM
for blocks, meta, or the whole document. This should
help performance a bit (and it does, in my tests).
No more SingleQuoted, DoubleQuoted, InlineMath, DisplayMath.
This makes everything uniform and predictable, though it does
open up a difference btw lua filters and custom writers.
I tested this with the str.lua filter on MANUAL.txt, and
I could see no significant performance degradation.
Doing things this way will ease maintenance, as we won't
have to manually modify this module when types change.
@tarleb, do we really need special cases for things like
DoubleQuoted and InlineMath?
The code still allowed to pass an arbitrary number of arguments to the
filter function, as element properties were passed as function arguments
at some point. Now we only pass the element as the single arg, so the
code to handle multiple arguments is no longer necessary.
Changed markdown, rtf, and HTML-based templates accordingly.
This allows you to set `toc: true` in the metadata; this
previously produced strange results in some output formats.
Closes#2872.
For backwards compatibility, `toc` is still set to the
toc contents. But it is recommended that you update templates
to use `table-of-contents` for the toc contents and `toc`
for a boolean flag.
This adds the required attributes to the temporary styles,
and also replaces existing language attributes in styles.xml.
Support for lang attributes on Div and Span has also been
added.
Closes#1667.
Now these functions return a pair of a reader/writer and an
Extensions, instead of building the extensions into the
reader/writer. The calling code must explicitly set
readerExtensions or writerExtensions using the Extensions
returned.
The point of the change is to make it possible for the
calling code to determine what extensions are being used.
See #3659.
If the metadata field is all on one line, we try to interpret
it as Inlines, and only try parsing as Blocks if that fails.
If it extends over one line (including possibly the `|` or
`>` character signaling an indented block), then we parse as
Blocks.
This was motivated by some German users finding that
date: '22. Juin 2017'
got parsed as an ordered list.
Closes#3755.
[API change]
The EPUB writer now takes its EPUB subdirectory from this option.
Also added `PandocEpubSubdirectoryError` to `PandocError`.
This is raised if the EPUB subdirectory is not all ASCII
alphanumerics.
See #3720.
See #3720.
We now put all EPUB related content in an EPUB/ subdirectory
by default (later this will be configurable).
mimetype
META-INF/
com.apple.ibooks.display-options.xml
container.xml
EPUB/ <<--configurable-->>
fonts/ <<--static-->>
font.otf
media/ <<--static-->>
cover.jpg
fig1.jpg
styles/ <<--static-->>
stylesheet.css
content.opf
toc.ncx
text/ <<--static-->>
ch001.xhtml
The readers previously assumed that CRs had been filtered
from the input. Now we strip the CRs in the readers themselves,
before parsing. (The point of this is just to simplify the
parsers.)
Shared now exports a new function `crFilter`. [API change]
And `tabFilter` no longer filters CRs.
Using the registry directly instead of a custom table is cleaner and
more efficient. The performance improvement is especially noticable when
filtering on frequent elements like Str.
Note that if the table has a first page header and a
continuation page header, the notes will appear only
on the first occurrence of the header.
Closes#2378.
Formerly tracing was just log messages with a DEBUG log
level. We now make these things independent. Tracing
can be turned on or off in PandocMonad using `setTrace`;
it is independent of logging.
* Removed `DEBUG` from `Verbosity`.
* Removed `ParserTrace` from `LogMessage`.
* Added `trace`, `setTrace` to `PandocMonad`.
* New module Text.Pandoc.Readers.Vimwiki, exporting readVimwiki [API change].
* New input format `vimwiki`.
* New data file, `data/vimwiki.css`, for displaying the HTML produced by this reader and pandoc's HTML writer in the style of vimwiki's own HTML export.
This will ensure that we only need to update these in one place.
(Currently, for example, the mathjax URL is used in both
App and trypandoc.)
Closes#3685.
This is a thin wrapper around mathjax that makes math look better
on revealjs.
See https://github.com/hakimel/reveal.js/#mathjax
We do this by setting the 'mathjax' boolean variable and
using it in the revealjs template. Also, for revealjs
and mathjax, we don't assign the usual thing to the 'math'
variable, since it's handled by mathjax config.
Closes#3743.
* Added `MissingCharacter` to `LogMessage` in Text.Pandoc.Logging.
* Parse the (xe)latex log for missing character warnings and issue
the warning.
Closes#3742.
This makes it sensitive to proper verbosity settings.
(It is now treated as INFO rather than WARNING, so one
doesn't get these messages for creation of tmp images
while making a pdf.)
API changes:
* Removed extractMediaBag from Text.Pandoc.MediaBag.
* Added Extracting as constructor for LogMessage.
Note that as a result of this change, the following,
which formerly produced a header with two lines separated
by a line break, will now produce a header followed by a
paragraph:
# Hi\
there
This may affect some existing documents that relied on
this undocumented and unintended behavior.
This change makes pandoc more consistent with other
Markdown implementations, and with itself (since the two-space
version of a line break doesn't work inside ATX headers, and
neither version works inside Setext headers).
Closes#3730.
* XML.toEntities: changed type to Text -> Text.
* Shared.tabFilter -- fixed so it strips out CRs as before.
* Modified writers to take Text.
* Updated tests, benchmarks, trypandoc.
[API change]
Closes#3731.
Hi, I don't know haskell so possibly this is wrong, but DOCX stores keywords in cp:keywords in core.xml, and this should be easy to add from the pandoc metadata (I copy and paste the author code). As far as I can tell (no clear documentation, just a few refs), keywords should be separated with a comma.
It is already in the eitherBlockOrInlineTag list, and
should be both places.
Closes#3717.
Note: the result of this change is that there will be
p tags around the whole paragraph. That is the right
result, because the `button` tags are treated as inline
HTML here, and the whole chunk of text is a Markdown
paragraph.
Currently we only handle the form `0.9\linewidth`.
Anything else would have to be converted to a percentage,
using some kind arbitrary assumptions about line widths.
See #3709.
The Emacs default is to include tags in the headline when exporting.
Instead of just empty spans, which contain the tag name as attribute,
tags are rendered as small caps and wrapped in those spans.
Non-breaking spaces serve as separators for multiple tags.
Babel result blocks can have block attributes like captions and names.
Result blocks with attributes were not recognized and were parsed as
normal blocks without attributes.
Fixes: #3706
Until now, org-ref cite keys included special characters also at the
end. This caused problems when citations occur right before colons or
at the end of a sentence.
With this change, all non alphanumeric characters at the end of a cite
key are ignored.
This also adds `,` to the list of special characters that are legal
in cite keys to better mirror the behaviour of org-export.
By not checking for the end condition before the first parse, the
parser was applied too often, consuming too much of the input.
This fixes the behaviour of
`testStringWith (many1Till (oneOf "ab") (string "aa")) "aaa"`
which before incorrectly returned `Right "a"`. With this change, it
instead correctly fails with `Left (PandocParsecError ...)` because it
is not able to parse at least one occurence of `oneOf "ab"` that is
not `"aa"`.
Note that this only affects `many1Till p end` where `p` matches on a
prefix of `end`.
With `--reference-location` of `section` or `block`, pandoc
will now repeat references that have been used in earlier
sections.
The Markdown reader has also been modified, so that *exactly*
repeated references do not generate a warning, only
references with the same label but different targets.
The idea is that, with references after every block,
one might want to repeat references sometimes.
Closes#3701.
Emacs parses org documents into a tree structure, which is then
post-processed during exporting. The reader is changed to do the same,
turning the document into a single tree of headlines starting at
level 0.
Fixes: #3695
- Export `inEm` from ImageSize [API change].
- Change `showFl` and `show` instance for `Dimension` so
extra decimal places are omitted.
- Added `Em` as a constructor of `Dimension` [API change].
- Allow `em`, `cm`, `in` to pass through without conversion
in HTML, LaTeX.
Closes#3450.
This is now the default for pandoc's Markdown.
It allows whitespace between the two parts of a
reference link: e.g.
[a] [b]
[b]: url
This is now forbidden by default.
Closes#2602.
This is a verison of parseFromString specialied to
ParserState, which resets stateLastStrPos at the end.
This is almost always what we want.
This fixes a bug where `_hi_` wasn't treated as emphasis in
the following, because pandoc got confused about the
position of the last word:
- [o] _hi_
Closes#3690.
We also export the set of known `schemes`.
The new function replaces the function of the same name
from `Network.URI`, as the latter did not check whether a scheme is
well-known. E.g. MediaWiki wikis frequently feature pages with names
like `User:John`. These links were interpreted as URIs, thus turning
internal links into global links. This is prevented by also checking
whether the scheme of a URI is frequently used (i.e. is IANA registered
or an otherwise well-known scheme).
Fixes: #2713
Update set of well-known URIs from IANA list
All official IANA schemes (as of 2017-05-22) are included in the set of
known schemes. The four non-official schemes doi, isbn, javascript, and
pmid are kept.
SEARCHPATH is separated by the usual character,
depending on OS (: on unix, ; on windows).
Note: This does not yet work for PDF output, because the
routine that creates PDFs runs outside PandocMonad.
(This has to do with its use of inTemporaryDirectory and
its interaction with our exceptions.)
The best solution would be to figure out how to move the
PDF creation routines into PandocMonad. Second-best,
just pass an extra parameter in?
See #852.
Closes#3629.
However, there is still room for improvement.
`@import` with following media declaration is not
handled.
Also `@import` with a simple filename (rather than
`url(...)` is not handled.
Parsing of smart quotes and special characters can either be enabled via
the `smart` language extension or the `'` and `-` export options. Smart
parsing is active if either the extension or export option is enabled.
Only smart parsing of special characters (like ellipses and en and em
dashes) is enabled by default, while smart quotes are disabled.
This means that all smart parsing features will be enabled by adding the
`smart` language extension. Fine-grained control is possible by leaving
the language extension disabled. In that case, smart parsing is
controlled via the aforementioned export OPTIONS only.
Previously, all smart parsing was disabled unless the language extension
was enabled.
This fixes a regression that was introduced when `--extract-media`
was generalized to work with any input format. We were getting
two versions of each image extracted from a docx, one with a hash,
one with the original filename, though only the hash one was used.
This patch restores the original behavior (using the original
filename).
Pointed out in comments on #3674. Thanks to @laperouse.
Support for the `#+INCLUDE:` file inclusion mechanism was added.
Recognized include types are *example*, *export*, *src*, and normal org
file inclusion. Advanced features like line numbers and level selection
are not implemented yet.
Closes: #3510
The grid table parsers for markdown and rst was combined into one single
parser, slightly changing parsing behavior of both parsers:
- The markdown parser now compactifies block content cell-wise: pure
text blocks in cells are now treated as paragraphs only if the cell
contains multiple paragraphs, and as plain blocks otherwise. Before,
this was true only for single-column tables.
- The rst parser now accepts newlines and multiple blocks in header
cells.
Closes: #3638
Supporting two completely different libraries for fetching
from URLs makes it difficult to trap errors, because of
different error types expected from the libraries.
There's no clear reason not to build with these https-capable
libraires.
If `--extract-media` is supplied with a non-binary input format,
pandoc will attempt to extract the contents of all linked images,
whether in local files, data: uris, or external uris.
They will be named based on the sha1 hash of the contents.
Closes#1583, #2289.
Notes:
- One thing that is slightly subideal with this commit is that
identical resources will be downloaded multiple times. To improve
this we could have mediabag store an original filename/url +
a new name.
- We might think about reusing some of this code, since more or less the
same thing is done in the Docx, EPUB, PDF writers (with slight
variations).
Previously we inadvertently interpreted indented HTML as
code blocks. This was a regression.
We now seek to determine the indentation level of the contents
of an HTML block, and (optionally) skip that much indentation.
As a side effect, indentation may be stripped off of raw
HTML blocks, if `markdown_in_html_blocks` is used. This
is better than having things interpreted as indented code
blocks.
Closes#1841.
* Fix keyval funtion: pandoc did not parse options in braces correctly. Additionally, dot, dash, and colon were no valid characters
* Add | as possible option value
* Improved code
This can happen e.g. with an @import of a google web font.
(What is imported is some CSS which contains an url reference
to the font itself.)
Also, allow unescaped pipe (|) in URL.
This is intended to help with #3629, but it doesn't seem to
work.
The parsing functions `tableWith` and `gridTableWith` are generalized to
work with more parsers. The parser state only has to be an instance of
the `HasOptions` class instead of requiring a concrete type. Block
parsers are required to return blocks wrapped into a monad, as this
makes it possible to use parsers returning results wrapped in `Future`s.
Previously the Markdown writer would sometimes create links where there
were none in the source. This is now avoided by selectively escaping bracket
characters when they occur in a place where a link might be created.
Closes#3619.
Ensure that we do not generate reference links
whose labels differ only by case.
Also allow implicit reference links when the link
text and label are identical up to case.
Closes#3615.
The implicitly defined global filter (i.e. all element filtering
functions defined in the global lua environment) is used if no filter is
returned from a lua script. This allows to just write top-level
functions in order to define a lua filter. E.g
function Emph(elem) return pandoc.Strong(elem.content) end
Attributes was written to behave much like a normal table, in order to
simplify working with it. However, all Attr containing elements were
changed to provide panflute-like accessors to Attr components, rendering
the previous approach unnecessary.
The `F` monads used for delayed evaluation of certain values in the
Markdown and Org readers are based on a shared data type capturing the
common pattern of both `F` types.
Previously the LaTeX writer created invalid LaTeX
when `--listings` was specified and a code span occured
inside emphasis or another construction.
This is because the characters `%{}\` must be escaped
in lstinline when the listinline occurs in another
command, otherwise they must not be escaped.
To deal with this, adoping Michael Kofler's suggestion,
we always wrap lstinline in a dummy command `\passthrough`,
now defined in the default template if `--listings` is
specified. This way we can consistently escape the
special characters.
Closes#1629.
A single `read` function parsing pandoc-supported formats is added to
the module. This is simpler and more convenient than the previous method
of exposing all reader functions individually.
Writer helper functions were defined in the top-level Text.Pandoc
module. These functions are moved to the Writer submodule as to enable
reuse in other submodules.
Reader helper functions were defined in the top-level Text.Pandoc
module. These functions are moved to the Readers submodule as to enable
reuse in other submodules.
Extension parsing and processing functions were defined in the top-level
Text.Pandoc module. These functions are moved to the Extensions
submodule as to enable reuse in other submodules.
Meta elements that are treated as lua tables (i.e. MetaList,
MetaInlines, MetaBlocks, and MetaMap), are no longer wrapped in an
additional table but simply marked via a metatable. This allows
treating those meta values just like normal tables, while still making
empty elements of those values distinguishable.
We now issue `<div class="line-block">` and include a
default definition for `line-block` in the default
templates, instead of hard-coding a `style` on the
div.
Closes#1623.
A figure with two subfigures turns into two pandoc
figures; the subcaptions are used and the main caption
ignored, unless there are no subcaptions.
Closes#3577.
Source block parameter names are no longer prefixed with *rundoc*. This
was intended to simplify working with the rundoc project, a babel
runner. However, the rundoc project is unmaintained, and adding those
markers is not the reader's job anyway.
The original language that is specified for a source element is now
retained as the `data-org-language` attribute and only added if it
differs from the translated language.
The line-numbering switch that can be given to source blocks (`-n` with
an start number as an optional parameter) is parsed and translated to a
class/key-value combination used by highlighting and other readers and
writers.
Previously we always added an empty div before the list
item, but this created problems with spacing in tight
lists. Now we do this:
If the list item contents begin with a Plain block,
we modify the Plain block by adding a Span around
its contents.
Otherwise, we add a Div around the contents of the
list item (instead of adding an empty Div to the
beginning, as before).
Closes#3596.
Element attributes are pushed to the stack via the `Attributes`
function. `Attributes` creates an Attr like triple, but the triple also
allows table-like access to key-value pairs.
Lua string are used to represent nullary data constructors. The previous
table-based representation was based on the JSON serialization, but can
be simplified. This also matches the way those arguments are passed to
custom writers.
The lua filters and custom lua writer system defined very similar
StackValue instances for strings and tuples. These instance definitions
are extracted to a separate module to enable sharing.
Native lua booleans and strings are used to represent MetaBool and
MetaString values. This is more natural than the previous table-based
representation. The old lua representation can still be read back to
haskell, ensuring compatibility with the `pandoc.MetaBool` and
`pandoc.MetaString` lua constructors.
Filtering functions take element components as arguments instead of the
whole block elements. This resembles the way elements are handled in
custom writers.
Instead of taking the whole inline element, forcing users to destructure it
themselves, the components of the elements are passed to the filtering
functions.
