Div's are difficult to translate into org syntax, as there are multiple
div-like structures (drawers, special blocks, greater blocks) which all
have their advantages and disadvantages. Previously pandoc would
use raw HTML to preserve the full div information; this was rarely
useful and resulted in visual clutter. Div-rendering was changed to
discard the div's classes and key-value pairs if there is no natural way
to translate the div into an org structure.
Closes: #3771
Previously pandoc would sometimes combine two line blocks separated by blanks, and ignore trailing blank lines within the line block.
Test is checked to be consisted with http://rst.ninjs.org/
This change makes it possible to define a catch-all function using lua's
metatable lookup functionality.
function catch_all(el)
…
end
return {
setmetatable({}, {__index = function(_) return catch_all end})
}
A further effect of this change is that the map with filter functions
now only contains functions corresponding to AST element constructors.
Closes#3511.
Previously pandoc used the four-space rule: continuation paragraphs,
sublists, and other block level content had to be indented 4
spaces. Now the indentation required is determined by the
first line of the list item: to be included in the list item,
blocks must be indented to the level of the first non-space
content after the list marker. Exception: if are 5 or more spaces
after the list marker, then the content is interpreted as an
indented code block, and continuation paragraphs must be indented
two spaces beyond the end of the list marker. See the CommonMark
spec for more details and examples.
Documents that adhere to the four-space rule should, in most cases,
be parsed the same way by the new rules. Here are some examples
of texts that will be parsed differently:
- a
- b
will be parsed as a list item with a sublist; under the four-space
rule, it would be a list with two items.
- a
code
Here we have an indented code block under the list item, even though it
is only indented six spaces from the margin, because it is four spaces
past the point where a continuation paragraph could begin. With the
four-space rule, this would be a regular paragraph rather than a code
block.
- a
code
Here the code block will start with two spaces, whereas under
the four-space rule, it would start with `code`. With the four-space
rule, indented code under a list item always must be indented eight
spaces from the margin, while the new rules require only that it
be indented four spaces from the beginning of the first non-space
text after the list marker (here, `a`).
This change was motivated by a slew of bug reports from people
who expected lists to work differently (#3125, #2367, #2575, #2210,
#1990, #1137, #744, #172, #137, #128) and by the growing prevalance
of CommonMark (now used by GitHub, for example).
Users who want to use the old rules can select the `four_space_rule`
extension.
* Added `four_space_rule` extension.
* Added `Ext_four_space_rule` to `Extensions`.
* `Parsing` now exports `gobbleAtMostSpaces`, and the type
of `gobbleSpaces` has been changed so that a `ReaderOptions`
parameter is not needed.
Previously only `[-@roe]` (with brackets) was recognized as
suppress-author, and `-@roe` was treated the same as `@roe`.
Closesjgm/pandoc-citeproc#237.
We now allow default output to stdout when it can be
determined that the output is being piped. (On Windows,
as mentioned before, this can't be determined.)
Using '-o -' forces output to stdout regardless.
Acronyms are not resolved by the reader, but acronym and glossary information is put into attributes on Spans so that they can be processed in filters.
Previously, for binary formats, output to stdout was disabled
unless we could detect that the output was being piped (and not
sent to the terminal). Unfortunately, such detection is not
possible on Windows, leaving windows users no way to pipe binary
output. So we have changed the behavior in the following way:
* If the -o option is not used, binary output is never sent
to stdout by default; instead, an error is raised.
* IF '-o -' is used, binary output is sent to stdout, regardless
of whether it is being piped. This works on Windows too.
Instead, just try running it and raise the exception if it
isn't found at that point.
This improves things for users of Cygwin on Windows, where
the executable won't be found by `findExecutable` unless
`.exe` is added.
The same exception is raised as before, but at a later
point.
Closes#3819.
The structure expected is:
<div class="columns">
<div class="column" width="40%">
contents...
</div>
<div class="column" width="60%">
contents...
</div>
</div>
Support has been added for beamer and all HTML slide formats.
Closes#1710.
Note: later we could add a more elegant way to create
this structure in Markdown than to use raw HTML div elements.
This would come for free with a "native div syntax" (#168).
Or we could devise something specific to slides
@ is commonly used in macros using `\makeatletter`.
Ideally we'd make the tokenizer sensitive to `\makeatletter`
and `\makeatother`, but until then this seems a good change.
Stack instances for common data types are now provides by hslua. The
instance for Either was useful only for a very specific case; the
function that was using the `ToLuaStack Either` instance was rewritten
to work without it.
Closes: #3805
Raw table accessing functions never call back into haskell, which allows
the compiler to use more aggressive optimizations. This improves lua
filter performance considerably (⪆5% speedup).
We assume that comments are defined as parsed by the
docx reader:
I want <span class="comment-start" id="0" author="Jesse Rosenthal"
date="2016-05-09T16:13:00Z">I left a comment.</span>some text to
have a comment <span class="comment-end" id="0"></span>on it.
We assume also that the id attributes are unique and properly
matched between comment-start and comment-end.
Closes#2994.
Previously they would be transmitted to the template without
any escaping.
Note that `--M title='*foo*'` yields a different result from
---
title: *foo*
---
In the latter case, we have emphasis; in the former case, just
a string with literal asterisks (which will be escaped
in formats, like Markdown, that require it).
Closes#3792.
* readDataFile, readDefaultDataFile, getReferenceDocx,
getReferenceODT have been removed from Shared and
moved into Class. They are now defined in terms of
PandocMonad primitives, rather than being primitve
methods of the class.
* toLang has been moved from BCP47 to Class.
* NoTranslation and CouldNotLoudTranslations have
been added to LogMessage.
* New module, Text.Pandoc.Translations, exporting
Term, Translations, readTranslations.
* New functions in Class: translateTerm, setTranslations.
Note that nothing is loaded from data files until
translateTerm is used; setTranslation just sets the
language to be used.
* Added two translation data files in data/translations.
* LaTeX reader: Support `\setmainlanguage` or `\setdefaultlanguage`
(polyglossia) and `\figurename`.
We bypass the commonmark writer from cmark and construct our
own pipe tables, with better results. (Note also that cmark-gfm
currently doesn't support rendering table nodes; see
kivikakk/cmark-gfm-hs#3.)
when enabled (as with gfm). Note: because of limitations in
cmark-gfm, which will hopefully soon be corrected, this currently
gives an error on Tables.
Also properly support `--wrap=none`.
We no longer have a separate readGFM and writeGFM;
instead, we'll use readCommonMark and writeCommonMark
with githubExtensions.
It remains to implement these extensions conditionally.
Closes#3841.
This uses bindings to GitHub's fork of cmark, so it should parse
gfm exactly as GitHub does (excepting certain postprocessing
steps, involving notifications, emojis, etc.).
* Added Text.Pandoc.Readers.GFM (exporting readGFM)
* Added Text.Pandoc.Writers.GFM (exporting writeGFM)
* Added `gfm` as input and output forma
Note that tables are currently always rendered as HTML
in the writer; this can be improved when CMarkGFM supports
tables in output.
Also, fix regular macros so they're expanded at the
point of use, and NOT also the point of definition.
`\let` macros, by contrast, are expanded at the
point of definition. Added an `ExpansionPoint`
field to `Macro` to track this difference.
We used to parse paragraphs styled with "HeadingN" as "nth-level
header." But if a document has a custom style named "Heading0", this
will produce a 0-level header, which shouldn't exist. We only parse
this style if N>0. Otherwise we treat it as a normal style name, and
follow its dependencies, if any.
Closes#3830.
We previously did this only with raw blocks, on the assumption
that math environments would always be raw blocks. This has changed
since we now parse them as inline environments.
Closes#3816.
Thus, a span with attribute 'foo' gets written to HTML5
with 'data-foo', so it is valid HTML5.
HTML4 is not affected.
This will allow us to use custom attributes in pandoc without
producing invalid HTML.
It is no longer necessary, since the rawLaTeXBlock parser
will parse macro definitions.
This also avoids the need for a separate latexMacro parser
in the Markdown reader.
Fixed applyMacros so that it operates on the whole
string, not just the first token!
Don't remove macro definitions from the output,
even if Ext_latex_macros is set, so that macros will
be applied. Since they're only applied to math in
Markdown, removing the macros can have bad effects.
Even for math macros, keeping them should be harmless.
An unknown command at the beginning of the line that could
be either block or inline is treated as block if we have
a sequence of block commands followed by a newline or a
`\startXXX` command (which might start a raw ConTeXt environment).
Added TikiWiki reader, including tests and documentation.
It's probably not *complete*, but it works pretty well, handles all
the basics (and some not-so-basics).
We may want to think of some kind of graceful fallback, but
the present behavior has the advantage of forcing people to
update scripts when updating to pandoc 2.0.
See #3786.
This rewrite is primarily motivated by the need to
get macros working properly. A side benefit is that the
reader is significantly faster (27s -> 19s in one
benchmark, and there is a lot of room for further
optimization).
We now tokenize the input text, then parse the token stream.
Macros modify the token stream, so they should now be effective
in any context, including math. Thus, we no longer need the clunky
macro processing capacities of texmath.
A custom state LaTeXState is used instead of ParserState.
This, plus the tokenization, will require some rewriting
of the exported functions rawLaTeXInline, inlineCommand,
rawLaTeXBlock.
* Added Text.Pandoc.Readers.LaTeX.Types (new exported module).
Exports Macro, Tok, TokType, Line, Column. [API change]
* Text.Pandoc.Parsing: adjusted type of `insertIncludedFile`
so it can be used with token parser.
* Removed old texmath macro stuff from Parsing.
Use Macro from Text.Pandoc.Readers.LaTeX.Types instead.
* Removed texmath macro material from Markdown reader.
* Changed types for Text.Pandoc.Readers.LaTeX's
rawLaTeXInline and rawLaTeXBlock. (Both now return a String,
and they are polymorphic in state.)
* Added orgMacros field to OrgState. [API change]
* Removed readerApplyMacros from ReaderOptions.
Now we just check the `latex_macros` reader extension.
* Allow `\newcommand\foo{blah}` without braces.
Fixes#1390.
Fixes#2118.
Fixes#3236.
Fixes#3779.
Fixes#934.
Fixes#982.
You can now have the following fields in your YAML metadata,
and it will be treated appropriately in the generated
EPUB.
```
ibooks:
version: 1.3.4
specified-fonts: false
ipad-orientation-lock: portrait-only
iphone-orientation-lock: landscape-only
binding: true
scroll-axis: vertical
```
This commit also fixes a regression in stylesheet paths.
GitHub has two Markdown modes, one for long-form documents like READMEs
and one for short things like issue coments. In issue comments, a line
break is treated as a hard line break. In README, wikis, etc., it is
treated as a space as in regular Markdown.
Since pandoc is more likely to be used to convert long-form
documents from GitHub Markdown, `-hard_line_breaks` is a better
default.
Closes#3594.
in Text.Pandoc.Lua. Also to pushPandocModule.
This change allows users to override pandoc.lua with a file
in their local data directory, adding custom functions, etc.
@tarleb, if you think this is a bad idea, you can revert this.
But in general our data files are all overridable.
Also, now we check before running walkM that the function
table actually does contain something relevant. E.g. if
your filter just defines Str, there's no need to run walkM
for blocks, meta, or the whole document. This should
help performance a bit (and it does, in my tests).
No more SingleQuoted, DoubleQuoted, InlineMath, DisplayMath.
This makes everything uniform and predictable, though it does
open up a difference btw lua filters and custom writers.
I tested this with the str.lua filter on MANUAL.txt, and
I could see no significant performance degradation.
Doing things this way will ease maintenance, as we won't
have to manually modify this module when types change.
@tarleb, do we really need special cases for things like
DoubleQuoted and InlineMath?
The code still allowed to pass an arbitrary number of arguments to the
filter function, as element properties were passed as function arguments
at some point. Now we only pass the element as the single arg, so the
code to handle multiple arguments is no longer necessary.
Changed markdown, rtf, and HTML-based templates accordingly.
This allows you to set `toc: true` in the metadata; this
previously produced strange results in some output formats.
Closes#2872.
For backwards compatibility, `toc` is still set to the
toc contents. But it is recommended that you update templates
to use `table-of-contents` for the toc contents and `toc`
for a boolean flag.
This adds the required attributes to the temporary styles,
and also replaces existing language attributes in styles.xml.
Support for lang attributes on Div and Span has also been
added.
Closes#1667.
Now these functions return a pair of a reader/writer and an
Extensions, instead of building the extensions into the
reader/writer. The calling code must explicitly set
readerExtensions or writerExtensions using the Extensions
returned.
The point of the change is to make it possible for the
calling code to determine what extensions are being used.
See #3659.
If the metadata field is all on one line, we try to interpret
it as Inlines, and only try parsing as Blocks if that fails.
If it extends over one line (including possibly the `|` or
`>` character signaling an indented block), then we parse as
Blocks.
This was motivated by some German users finding that
date: '22. Juin 2017'
got parsed as an ordered list.
Closes#3755.
[API change]
The EPUB writer now takes its EPUB subdirectory from this option.
Also added `PandocEpubSubdirectoryError` to `PandocError`.
This is raised if the EPUB subdirectory is not all ASCII
alphanumerics.
See #3720.
See #3720.
We now put all EPUB related content in an EPUB/ subdirectory
by default (later this will be configurable).
mimetype
META-INF/
com.apple.ibooks.display-options.xml
container.xml
EPUB/ <<--configurable-->>
fonts/ <<--static-->>
font.otf
media/ <<--static-->>
cover.jpg
fig1.jpg
styles/ <<--static-->>
stylesheet.css
content.opf
toc.ncx
text/ <<--static-->>
ch001.xhtml
The readers previously assumed that CRs had been filtered
from the input. Now we strip the CRs in the readers themselves,
before parsing. (The point of this is just to simplify the
parsers.)
Shared now exports a new function `crFilter`. [API change]
And `tabFilter` no longer filters CRs.
Using the registry directly instead of a custom table is cleaner and
more efficient. The performance improvement is especially noticable when
filtering on frequent elements like Str.
Note that if the table has a first page header and a
continuation page header, the notes will appear only
on the first occurrence of the header.
Closes#2378.
Formerly tracing was just log messages with a DEBUG log
level. We now make these things independent. Tracing
can be turned on or off in PandocMonad using `setTrace`;
it is independent of logging.
* Removed `DEBUG` from `Verbosity`.
* Removed `ParserTrace` from `LogMessage`.
* Added `trace`, `setTrace` to `PandocMonad`.
* New module Text.Pandoc.Readers.Vimwiki, exporting readVimwiki [API change].
* New input format `vimwiki`.
* New data file, `data/vimwiki.css`, for displaying the HTML produced by this reader and pandoc's HTML writer in the style of vimwiki's own HTML export.
This will ensure that we only need to update these in one place.
(Currently, for example, the mathjax URL is used in both
App and trypandoc.)
Closes#3685.
This is a thin wrapper around mathjax that makes math look better
on revealjs.
See https://github.com/hakimel/reveal.js/#mathjax
We do this by setting the 'mathjax' boolean variable and
using it in the revealjs template. Also, for revealjs
and mathjax, we don't assign the usual thing to the 'math'
variable, since it's handled by mathjax config.
Closes#3743.
* Added `MissingCharacter` to `LogMessage` in Text.Pandoc.Logging.
* Parse the (xe)latex log for missing character warnings and issue
the warning.
Closes#3742.
