Now RTL is turned and off by a general function, `withDirection`
wrapping `inlineToOpenXML` and `blockToOpenXML`. This acts according to
the `envRTL` variable. This means we can just set the environment at the
outset, and change the environment with `local` as need be.
Note that this requires making the `inlineToOpenXML` and
`blockToOpenXML` functions into wrappers around
primed-versions (`{inline,block}ToOpenXML`) where the real work takes
place.
In general, we want things that are either:
1. unchanging environment variables, or
2. environment variables that will change for a the scope of
a function and then pop back
to be in the reader monad. This is safer for (1), since we won't
accidentally change it, and easier for (2), since we can use `local`
instad of setting the old value and then resetting.
We keep the StateT monad for values that we will want to accumulate or
change and then use later.
We had to use this because we set the env, which means that setRTL
wouldn't do anything at the top level. We now don't set the env (it will
always be false at the outset), which means the toplevel setRTL will
work if necessary.
This will allow us to add text and paragraph properties depending on if
rtl is already set or not.
(It would probably be cleaner and safer to move the paraprops and
textprops to this part of the stack in the future.)
In AsciiDoc, you must use a special form of emphasis (double `__`)
for intraword emphasis. Pandoc was previously using this more
than necessary.
Closes#3068.
Backticks in verbatim environments are converted to
open-single-quotes. This change makes them appear as backticks. This
corresponds to how we treat `'' in verbatim environments (with
\textquotesingle{}).
Add --parts command line argument.
This only effects LaTeX writer, and only for non-beamer output formats.
It changes the output levels so the top level is 'part', the next
'chapter' and then into sections.
This tests for a min value >= 0.5. But we have a lower bound of 0.5 in
pandoc.cabal, so the test will always pass.
(If we bump the lower bound to 0.5.1, we can remove a conditional in the
HTML writer as well.)
Because time 1.4 is a boot library for GHC 7.8, we will support the
compatibility module as long as we support 7.8. But we should be clear
about when we will no longer need it.
We already lower-bound tagsoup at 0.13.7, which means we were always
running the compatibility layer (it was conditional on min value
0.13). Better to just use `lookupEntity` from the library directly, and
convert a string to a char if need be.
Instead, emit the alt text, emphasized. This accords with what
the ODT writer currently does.
The user will still get a warning about a nonexistent image,
but will no longer get a LaTeX crash.
Closes#3100.
Sections the `unnumbered` property should, as the name implies, be
excluded from the automatic numbering of section provided by some output
formats. The Pandoc convention for this is to add an "unnumbered" class
to the header. The reader treats properties as key-value pairs per
default, so a special case is added to translate the above property to a
class instead.
Closes#3095.
The `creator` option controls whether the creator meta-field should be
included in the final markup. Setting `#+OPTIONS: creator:nil` will
drop the creator field from the final meta-data output.
Org-mode recognizes the special value `comment` for this field, causing
the creator to be included in a comment. This is difficult to translate
to Pandoc internals and is hence interpreted the same as other truish
values (i.e. the meta field is kept if it's present).
The `email` option controls whether the email meta-field should be
included in the final markup. Setting `#+OPTIONS: email:nil` will drop
the email field from the final meta-data output.
The `author` option controls whether the author should be included in
the final markup. Setting `#+OPTIONS: author:nil` will drop the author
from the final meta-data output.
HTML-specific head content can be defined in `#+HTML_head` lines. They
are parsed as format-specific inlines to ensure that they will only show
up in HTML output.
LaTeX-specific header commands can be defined in `#+LaTeX_header` lines.
They are parsed as format-specific inlines to ensure that they will only
show up in LaTeX output.
The last meta-line of any given type is the significant line.
Previously the value of the first line was kept, even if more lines of
the same type were encounterd.
Previously we only used the first anchor span to affect header ids. This
allows us to use all the anchor spans in a header, whether they're
nested or not.
Along with 62882f97, this closes#3088.
Previously we always generated an id for headers (since they wouldn't
bring one from Docx). Now we let it use an existing one if
possible. This should allow us to recurs through anchor spans.
Previously, we would only be able to figure out internal links to a
header in a docx if the anchor span was empty. We change that to read
the inlines out of the first anchor span in a header.
This still leaves another problem: what to do if there are multiple
anchor spans in a header. That will be addressed in a future commit.
Pandoc and Org-mode use different programming language identifiers. An
additional translation between those identifiers is added to avoid
unexpected behavior. This fixes a problem where language specific
source code would sometimes be output as example code.
Org-mode treats links as document internal searches unless the link
target looks like a URL or file path, either relative or absolute. This
change ensures that this is always the case.
An Org-mode figure should be surrounded by blank lines. The figure
would be recognized regardless, but images in the following line would
unintentionally be treated as figures as well.
This enables dynamic styling on spans. It uses the same prefix as we
used on divs ("docx-style" for the moment). It does not yet inject the
style into styles.xml.
The functions `isElem` and `elemName` (defined in Docx/Util.hs) make the
code a lot cleaner than the original XML.Light functions, but they had
been used inconsistently. This puts them in wherever applicable.
Image sources as those in plain images, image links, or figures, must be
proper URIs or relative file paths to be recognized as images. This
restriction is now enforced for all image sources.
This also fixes the reader's usage of uncleaned image sources, leading
to `file:` prefixes not being deleted from figure
images (e.g. `[[file:image.jpg]]` leading to a broken image `<img
src="file:image.jpg"/>)
Thanks to @bsag for noticing this bug.
Previously these yielded strings of alternating Code and Space
elements; we now incorporate the spaces into the Code. Emphasis
etc. is still possible inside these.
Closes#3055.
Previously an unquoted attribute value in a table row
could cause parsing problems.
Fixes#3053 (well, proper rowspans and colspans aren't
created, but that's a bigger limitation with the current
Pandoc document model for tables).
In the latex parser when includes are processed, the text of the
included file is directly included into the parse stream. This caused
problems when there was an error in the included file (and the included
file was longer than the original file) as the error would be reported
at this position.
The error handling tries to display the line and position where the
error occured. It works by including a copy of the input and finding the
place in the input when given the position of the error. In the
previously described scenario, the input file would be the original
source file but the error position would be the position of the error in
the included file.
The fix is to not try to show the exact line when it would cause an
out-of-bounds error.
The starred variants don't exist.
This helps with part of #3058...it gets rid of the spurious *s.
But we still have numbers on the 4th and 5th level headers.
Previously blockquotes were used. Now a Div is used
with class `admonition` and (if relevant) one of the
following: `attention`, `caution`, `danger`, `error`,
`hint`, `important`, `note`, `tip`, `warning`.
`sidebar` is also put into a Div.
Note: This will change rendering of RST documents!
It should provide much more flexibility.
Closes#3031.
We now handle cell and row attributes, mostly by skipping
them. However, alignments are now handled properly.
Since in pandoc alignment is per-column, not per-cell, we
try to devine column alignments from cell alignments.
Table captions are also now parsed, and textile indicators
for thead and tfoot no longer cause parse failure. (However,
a row designated as tfoot will just be a regular row in pandoc.)
Previously, we had used the user-supplied date, if available, for Word's
document creation metadata. This could lead to weird results, as in
cases where the user post-dates a document (so the modification might be
prior to the creation). Here we use the actual computer time to set the
document creation.
Previously we parsed a list of dates, took the first one, and then
tested its year range. That meant that if the first one failed, we
returned nothing, regardless of what the others did. Now we test for
sanity before running `msum` over the list of Maybe values. Anything
failing the test will be Nothing, so will not be a candidate.
We only allow years between 1601 and 9999, inclusive. The ISO 8601
actually says that years are supposed to start with 1583, but MS Word
only allows 1601-9999. This should stop corrupted word files if the date
is out of that range, or is parsed incorrectly.
We want to avoid illegal dates -- in particular years with greater than
four digits. We attempt to parse series of digits first as `%Y%m%d`, then
`%Y%m`, and finally `%Y`.
Org rules for allowed characters before or after markup chars were not
checked for verbatim text. This resultet in wrong parsing outcomes of
if the verbatim text contained e.g. space enclosed markup characters as
part of the text (`=is_substr = True=`). Forcing the parser to update
the positions of allowed/forbidden markup border characters fixes this.
This fixes#3016.
Div blocks handling is changed to make the output look more like
idiomatic org mode:
- Div-wrapped content is output as-is if the div's attribute is the
null attribute.
- Div containers with an id but neither classes nor key-value pairs
are unwrapped and the id is added as an anchor.
- Divs with classes associated with greater block elements are
wrapped in a `#+BEGIN`...`#+END` block.
- The old behavior for Divs with more complex attributes is kept.
Some less-than-smart code required a pragma switching of overlapping
pattern warnings in order to compile seamlessly. Using view patterns
makes the code easier to read and also doesn't require overlapping
pattern checks to be disabled.
Handling of archived trees can be modified using the `arch` option.
Archived trees are either dropped, exported completely, or collapsed to
include just the header when the `arch` option is nil, non-nil, or
`headline`, respectively.
Comment trees were handled after parsing, as pattern matching on lists
is easier than matching on sequences. The new method of reading
documents as trees allows for more elegant subtree removal.
Emacs org-mode is based on outline-mode, which treats documents as trees
with headlines are nodes. The reader is refactored to parse into a
similar tree structure. This simplifies transformations acting on
document (sub-)trees.
We treat display math like block quotes, and apply FirstParagraph style
to paragraphs that follow them. These can be styled as the user
wishes. (But, when the user is using indentation, this allows for
paragraphs to continue after display math without indentation.)
Figure labels given as `#+LABEL: thelabel` are used as the ID of the
respective image. This allows e.g. the LaTeX to add proper `\label`
markup.
This fixes half of #2496 and #2999.
The link
`<foo>`_
should have `foo` as both its link text and its URL.
See RST spec at
<http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#embedded-uris-and-aliases>
"The reference text may also be omitted, in which case the URI will be
duplicated for use as the reference text. This is useful for relative
URIs where the address or file name is also the desired reference text:
See `<a_named_relative_link>`_ or `<an_anonymous_relative_link>`__
for details."
Closes Debian #828167 -- reported by Christian Heller.
Previously they were processed, very unintuitively, in R->L
order, so that `markdown-tex_math_dollars+tex_math_dollars`
had `tex_math_dollars` disabled.
Closes#2995.
We can't guarantee we'll convert every comment correctly, though we'll
do the best we can. This warns if the comment includes something other
than Para or Plain.
This adds simple track-changes comment parsing to the docx reader. It is
turned on with `--track-changes=all`. All comments are converted to
inlines, which can list some information. In the future a warning will
be added for comments with formatting that seems like it will be
excessively denatured.
Note that comments can extend across blocks. For that reason there are
two spans: `comment-start` and `comment-end`. `comment-start` will
contain the comment. `comment-end` will always be empty. The two will be
associated by a numeric id.
This is a lossy function for converting `[Block] -> [Inline]`. Its main
use, at the moment, is for docx comments, which can contain arbitrary
blocks (except for footnotes), but which will be converted to spans.
This is, at the moment, pretty useless for everything but the basic
`Para` and `Plain` comments. It can be improved, but the docx reader
should probably emit a warning if the comment contains more than this.
Previously we just passed all raw TeX through when MathJax
was used for HTML math. This passed through too much.
With this patch, only raw LaTeX environments that MathJax
can handle get passed through.
This patch also causes raw LaTeX environments to be treated
as math, when possible, with MathML and WebTeX output.
Closes#2758.
- `writerEmailObfuscation` in `defaultWriterOptions` is now
`NoObfuscation`
- the default for the command-line `--email-obfuscation` option is
now `none`.
Closes#2988.
Org mode allows arbitrary raw inlines ("export snippets" in Emacs
parlance) to be included as `@@format:raw foreign format text@@`.
Support for this features is added to the Org reader.
Org mode allows arbitrary raw inlines ("export snippets" in Emacs
parlance) to be included as `@@format:raw foreign format text@@`.
Support for this features is added to the Org writer.
A specification for an official Org-mode citation syntax was drafted by
Richard Lawrence and enhanced with the help of others on the orgmode
mailing list. Basic support for this citation style is added to the
reader.
This closes#1978.
Semicolons are used as special characters in citations syntax. This
ensures the correct parsing of Pandoc-style citations:
[prefix; @key; suffix]
Previously, parsing would have failed unless there was a space or other
special character as the last <prefix> character.
Parsing of emphasized text can be toggled using the `*` option. This
influences parsing of text marked as emphasized, strong, strikeout, and
underline. Parsing of inline math, code, and verbatim text is not
affected by this option.
The `OrgParserState` contained both an `orgStateMeta` and
`orgStateMeta'` field, the former for plain meta information and the
latter for F-monad wrapped meta info. The plain meta info is only used
to make `OrgParserState` an instance of the `HasMeta` class, which in
turn is never used in the reader. The (F Meta) version is hence renamed
to the "un-primed" version while the other one is dropped.
Some code was duplicated (copy-pasted) or placed in an inappropriate
module during the modularization refactoring. Those functions are moved
into a `Shared` module, as was originally intended but forgotten.
Better documentation of the respective functions is a positive
side-effect.
Org-mode version 9 usees a new syntax for export blocks. Instead of
`#+BEGIN_<FORMAT>`, where `<FORMAT>` is the format of the block's
content, the new format uses `#+BEGIN_export <FORMAT>` instead. Both
types are supported.
- Reorder functions, grouping related functions together.
- Demote simple functions to local functions if they are used just once.
- Rename and document functions to increase code readability.
- Fix handling of whitespace in blocks, allowing content to be indented
less then the block header.
Having a function starting with `parse` in a parsing library is overly
redundant. Let's use a nicer, shorter name more in line with the rest
of the library.
The *org-ref* package is an org-mode extension commonly used to manage
citations in org documents. Basic support for the `cite:citeKey` and
`[[cite:citeKey][prefix text::suffix text]]` syntax is added.
Inline parsing code is moved to a separate module. Parsers for block
starts are extracted as well, as those are used in the `endline` parser.
This is part of the Org-mode reader cleanup effort.
The Org-mode reader uses many functions defined in the
`Text.Pandoc.Parsing` utility module. Some of the functions are
overwritten with versions adapted to Org-mode idiosyncrasies. These
special functions, as well as the normal Pandoc versions, are combined
in a single module to increase the ease of use.
This leads to decoupling of Org-mode and Pandoc and hence to slightly
cleaner code. The downside is code-bloat due to repeated import/export
statements.
For the implementation of the Drawer element in the Org Writer, we make
use of a generic Block container with attributes. The presence of a
`drawer` class defines that the `Div` constructor is a drawer. The first
class defines the drawer name to use. The key-value list in the
attributes defines the keys to add inside the Drawer. Lastly, the list
of Block elements contains miscellaneous blocks elements to add inside
of the Drawer.
Signed-off-by: Albert Krewinkel <albert@zeitkraut.de>
The `ID` property is reserved for internal use by Org-mode and should
not be used. The `CUSTOM_ID` property is to be used instead, it is
converted to the `ID` property for certain export format.
The reader and writer erroneously used `ID`. This is corrected by using
`CUSTOM_ID` where appropriate.
This allows header attributes to be added to org documents in the form
of `:PROPERTIES:` drawers. All available attributes are stored as
key/value pairs. This reflects the way the org reader handles
`:PROPERTIES:` blocks.
This closes#1962.
Headers can have optional `:PROPERTIES:` drawers associated with them.
These drawers contain key/value pairs like the header's `id`. The
reader adds all listed pairs to the header's attributes; `id` and
`class` attributes are handled specially to match the way `Attr` are
defined.
This also changes behavior of how drawers of unknown type are handled.
Instead of including all unknown drawers, those are not read/exported,
thereby matching current Emacs behavior.
This closes#1877.
Arbitrary key-value pairs can be added to some block types using a
`#+ATTR_HTML` line before the block. Emacs Org-mode only includes these
when exporting to HTML, but since we cannot make this distinction here,
the attributes are always added.
The functionality is now supported for figures.
This closes#1906.
Additional state changes need to be made after a newline is parsed,
otherwise markup may not be recognized correctly.
This fixes a bug where markup after certain block-types would not be
recognized. E.g. `/emph/` in the following snippet was not parsed as
emphasized.
foo
# comment
/emph/
A parser state attribute was used to keep track of block attributes
defined in meta-lines. Global state is undesirable, so block attributes
are no longer saved as part of the parser state. Old functions and the
respective part of the parser state are removed.
Org-mode allows to specify export settings via `#+OPTIONS` lines.
Disabling simple sub- and superscripts is one of these export options,
this options is now supported.
The last fix for whitespace handling of inline LaTeX commands was
incorrect, preventing correct recognition of inline LaTeX commands which
contain spaces. This fix ensures that only trailing whitespace is cut
off.
The org-reader was droping space after unescaped LaTeX-style symbol
commands: `\ForAll \Auml` resulted in `∀Ä` but should give `∀ Ä`
instead. This seems to be because the LaTeX-reader treats the
command-terminating space as part of the command. Dropping the trailing
space from the symbol-command fixes this issue.
This fixes Org mode parsing of some corner cases regarding empty cells
and rows. Empty cells weren't parsed correctly, e.g. `|||` should be
two empty cells, but would be parsed as a single cell containing a pipe
character. Empty rows where parsed as alignment rows and dropped from
the output.
This fixes#2616.
Emacs Org-mode doesn't add any padding to table rows. The first
row (header or first body row) is used to determine the column count, no
other magic is performed.
The org reader was padding rows to the length of the longest table row.
This was done due to a misunderstanding of how Org handles tables. This
feature reflected how Org-mode handles tables when pressing <TAB>. The
Org exporter however, which is what the reader should implement, doesn't
do any of this. So this was a mis-feature that made the reader more
complex and reduced comparability. It was hence removed.
Previously, readDocx would error out if zip-archive failed. We change
the archive extraction step from `toArchive` to `toArchiveOrFail`, which
returns an Either value.
`moveTo` and `moveFrom` are track-changes tags that are used when a
block of text is moved in the document. We now recognize these tags and
treat them the same as `insert` and `delete`, respectively. So,
`--track-changes=accept` will show the moved version, while
`--track-changes=reject` will show the original version.
We are now more forgiving about parsing invalid HTML with
unescaped `&` as raw HTML. (Previously any unescaped `&`
would cause pandoc not to recognize the string as raw HTML.)
Closes#2410.
Partially addresses #2813.
This isn't perfect, because now the hypertarget is in the
wrong place -- when you link to the figure, the screen
is positioned with the caption at the top, and most of
the figure off screen.
So this needs a bit more tweaking.
This was a regression, with the rewrite of `htmlInBalanced`
(from `Text.Pandoc.Readers.HTML`) in 1.17.
It caused newlines to be omitted in raw HTML blocks.
Closes#2804.
Add a `\strut` after `\crlf` before space.
Closes#2744, #2745. Thanks to @c-foster.
This uses the fix suggested by @c-foster.
Mid-line spaces are still not supported, because of limitations
of the Markdown parser.
Some word functions -- especially graphics -- give various choices for
content so there can be backwards compatibility. This follows the
largely undocumented feature by working through the choices until we
find one that works.
Note that we had to split out the processing of child elems of runs into
a separate function so we can recurse properly. Any processing of an
element *within* a run (other than a plain run) should go into
`childElemToRun`.
Traditionally pandoc operates on multiple files by first concetenating
them (around extra line breaks) and then processing the joined file. So
it only parses a multi-file document at the document scope. This has the
benefit that footnotes and links can be in different files, but it also
introduces a couple of difficulties:
- it is difficult to join files with footnotes without some sort of
preprocessing, which makes it difficult to write academic documents
in small pieces.
- it makes it impossible to process multiple binary input files, which
can't be catted.
- it makes it impossible to process files from different input
formats.
This commit introduces alternative method. Instead of catting the files
first, it parses the files first, and then combines the parsed
output. This makes it impossible to have links across multiple files,
and auto-identified headers won't work correctly if headers in multiple
files have the same name. On the other hand, footnotes across multiple
files will work correctly and will allow more freedom for input formats.
Since ByteStringReaders can currently only read one binary file, and
will ignore subsequent files, we also changes the behavior to
automatically parse before combining if using the ByteStringReader. If
we use one file, it will work as normal. If there is more than one file
it will combine them after parsing (assuming that the format is the
same).
Note that this is intended to be an optional method, defaulting to
off. Turn it on with `--file-scope`.
In order to be able to collect warnings during parsing, we add a state
monad transformer to the D monad. At the moment, this only includes a
list of warning strings (nothing currently triggers them, however). We
use StateT instead of WriterT to correspond more closely with the
warnings behavior in T.P.Parsing.
The docx reader used to use a Modifiable typeclass to combine both
Blocks and Inlines. But all the work was in the inlines. So most of the
generality was wasted, at the expense of making the code harder to
understand. This gets rid of the generality, and adds functions for
Blocks and Inlines. It should be a bit easier to work with going forward.
We currently treat all memoir templates as books. This means that pandoc
will infer the `--chapters` argument, even if the `article` iption is
set for memoir.
This commit makes pandoc treats the document as an article if there is
an article option (i.e., `\documentclass[12pt,article]{memoir}`).
Note that this refactors out the parsec parsers for document class and
options, to make it a little clearer what's going on.
Previously smart quotes were incorrect in the following:
'$\neg(x \in x)$'.
(because of the following period). This commit fixes the problem,
which was introduced by commit 4229cf2d92.
This gives better results when people write e.g. `\TeX{}` in Markdown.
\TeX{} and \LaTeX{}
now works as expected with `pandoc -f markdown -t latex`.
Closes#2687.
Change types of divs.
From Docbook "sect#" and "simplesect" to "level#" and
"section."
Add tests.
Add mention of TEI to README.
Small changes to TEI writer.
The convention used by pandoc for figures is to mark them by prefixing
the name with "fig:". The org reader failed to do this if a figure had
no name. The test for this was broken as well.
This fixes#2643.
- Text.Pandoc.XML.fromEntities: handle entities without a
semicolon. Always lookup character references with the
trailing ';', even if it wasn't present. And never add
it when looking up numerical entities. (This is what
tagsoup seems to require.)
- Text.Pandoc.Parsing.characterReference: Always lookup
character references with the trailing ';', and leave off
the ';' when looking up numerical entities.
This fixes a regression for e.g. `⟨`.
Continue scanning for comment subtrees beyond only the first block.
Note to self: when writing an recursive function, don't forget to, you
know, actually recurse.
Shout to @mrvdb for noticing this.
This fixes#2628.
The reader previously did allow this, following redcloth,
which happily parses
Html blocks can be <div>inlined</div> as well.
as
<p>Html blocks can be <div>inlined</div> as well.</p>
This is invalid HTML, and this kind of thing can lead
to parsing problems (stack overflows) as well. So this
commit undoes this behavior. The above sample now produces;
<p>Html blocks can be</p>
<div>
<p>inlined</p>
</div>
<p>as well.</p>
+ Start cell on new line unless it's a single Para or Plain.
+ For single Para or Plain, insert a space after the `|` to
avoid problems when the text begins with a character like
`-`.
Closes#2604, closes#2606.
If `geometry` has no value, but `margin-left`, `margin-right`,
`margin-top`, and/or `-margin-bottom` are given, a default value
for `geometry` is created from these.
Note that these variables already affect PDF production via HTML5
with wkhtmltopdf.
Variables margin-top, margin-bottom, margin-left, margin-right.
Setting them with css inside @page doesn't seem to work, at least
with the released wkhtmltopdf.
* Added `thanks` variable
* Use `parskip.sty` when `indent` isn't set (fall
back to using `setlength` as before if `parskip.sty`
isn't available).
* Use `biblio-style` with biblatex.
* Added `biblatexoptions` variable.
* Added `section-titles` variable (defaults to true)
to enable/suppress section title pages in beamer
slide shows.
* Moved beamer themes after fonts, so that themes can
change fonts. (Previously the fonts set were being
clobbered by lmodern.sty.)
`file:filename` rather than `file://./filename`.
I think this is right; it matches what we had before
with people actually using the ICML writer, and seems
to match examples in the spec. I don't
have a copy of InDesign I can test on, though.
@DigitalPublishingToolkit and @mb21, can you have
a look?
