Code blocks that are not marked as a language supported by Jira are
rendered as preformatted text with `{noformat}` blocks.
Fixes: tarleb/jira-wiki-markup#4
[API change]
These were only exported for testing, which seems the
wrong thing to do. They don't belong in the public
API and are not really usable as they are, without access
to the Tok type which is not exported.
Removed the tokenize/untokenize roundtrip test.
We put a quickcheck property in the comments which
may be used when this code is touched (if it is).
[API change] This affects `readFile`, `getContents`, `writeFileWith`,
`writeFile`, `putStrWith`, `putStr`, `putStrLnWith`, `putStrLn`.
`hPutStrWith`, `hPutStr`, `hPutStrLnWith`, `hPutStrLn`, `hGetContents`.
This avoids the need to uselessly create a linked list of characters
when emiting output.
The org-ref syntax allows to list multiple citations separated by comma.
This fixes a bug that accepted commas as part of the citation id, so all
citation lists were parsed as one single citation.
Fixes: #7101
..and add new definitions isomorphic to xml-light's, but with
Text instead of String. This allows us to keep most of the code in
existing readers that use xml-light, but avoid lots of unnecessary
allocation.
We also add versions of the functions from xml-light's
Text.XML.Light.Output and Text.XML.Light.Proc that operate
on our modified XML types, and functions that convert
xml-light types to our types (since some of our dependencies,
like texmath, use xml-light).
Update golden tests for docx and pptx.
OOXML test: Use `showContent` instead of `ppContent` in `displayDiff`.
Docx: Do a manual traversal to unwrap sdt and smartTag.
This is faster, and needed to pass the tests.
Benchmarks:
A = prior to 8ca191604d (Feb 8)
B = as of 8ca191604d (Feb 8)
C = this commit
| Reader | A | B | C |
| ------- | ----- | ------ | ----- |
| docbook | 18 ms | 12 ms | 10 ms |
| opml | 65 ms | 62 ms | 35 ms |
| jats | 15 ms | 11 ms | 9 ms |
| docx | 72 ms | 69 ms | 44 ms |
| odt | 78 ms | 41 ms | 28 ms |
| epub | 64 ms | 61 ms | 56 ms |
| fb2 | 14 ms | 5 ms | 4 ms |
* Modified the Doc parser to skip leading blank lines. This fixes
parsing of documents which start with multiple blank lines.
(#7095)
* Prevent URLs within link aliases to be treated as autolinks.
(#6944)
Fixes: #7095Fixes: #6944
This exports functions that uses xml-conduit's parser to
produce an xml-light Element or [Content]. This allows
existing pandoc code to use a better parser without
much modification.
The new parser is used in all places where xml-light's
parser was previously used. Benchmarks show a significant
performance improvement in parsing XML-based formats
(especially ODT and FB2).
Note that the xml-light types use String, so the
conversion from xml-conduit types involves a lot
of extra allocation. It would be desirable to
avoid that in the future by gradually switching
to using xml-conduit directly. This can be done
module by module.
The new parser also reports errors, which we report
when possible.
A new constructor PandocXMLError has been added to
PandocError in T.P.Error [API change].
Closes#7091, which was the main stimulus.
These changes revealed the need for some changes
in the tests. The docbook-reader.docbook test
lacked definitions for the entities it used; these
have been added. And the docx golden tests have been
updated, because the new parser does not preserve
the order of attributes.
Add entity defs to docbook-reader.docbook.
Update golden tests for docx.
Mmny of our tests require running the pandoc
executable. This is problematic for a few different reasons.
First, cabal-install will sometimes run the test suite
after building the library but before building the executable,
which means the executable isn't in place for the tests.
One can work around that by first building, then building
and running the tests, but that's fragile. Second,
we have to find the executable. So far, we've done that
using a function findPandoc that attempts to locate it
relative to the test executable (which can be located
using findExecutablePath). But the logic here is delicate
and work with every combination of options.
To solve both problems, we add an `--emulate` option to
the `test-pandoc` executable. When `--emulate` occurs
as the first argument passed to `test-pandoc`, the
program simply emulates the regular pandoc executable,
using the rest of the arguments (after `--emulate`).
Thus,
test-pandoc --emulate -f markdown -t latex
is just like
pandoc -f markdown -t latex
Since all the work is done by library functions,
implementing this emulation just takes a couple lines
of code and should be entirely reliable.
With this change, we can test the pandoc executable
by running the test program itself (locatable using
findExecutablePath) with the `--emulate` option.
This removes the need for the fragile `findPandoc`
step, and it means we can run our integration tests
even when we're just building the library, not the
executable.
Part of this change involved simplifying some complex
handling to set environment variables for dynamic
library paths. I have tested a build with
`--enable-dynamic-executable`, and it works, but
further testing may be needed.
This reverts commit 6efd3460a7.
Since this extension is designed to be used with
GitHub markdown (gfm), we need to implement the parser
as a commonmark extension (commonmark-extensions),
rather than in pandoc's markdown reader. When that is
done, we can add it here.
Additional pipe chars, used to separate "action" state from "no further
action" states, are ignored. E.g., for the following sequence, both
`DONE` and `FINISHED` are states with no further action required.
#+TODO: UNFINISHED | DONE | FINISHED
Previously, parsing of the todo sequence failed if multiple pipe chars
were included.
Closes: #7014
* Replace org-mode’s verbatim from code to codeWith.
This adds the `"verbatim"` class so that exporters can apply a specific
style on it. For instance, it will be possible for HTML to add a CSS
rule for code + verbatim class.
* Alter test for org-mode’s verbatim change.
See previous commit for further detail on the new implementation.
The Div wrapper of code blocks with captions now has the class
"captioned-content". The caption itself is added as a Plain block
inside a Div of class "caption". This makes it easier to write filters
which match on captioned code blocks. Existing filters will need to be
updated.
Closes: #6977
Note that the multirow package is needed for rowspans.
It is included in the latex template under a variable,
so that it won't be used unless needed for a table.
Docbook reader produces a `Div` with `title` class for `<title>` element
within an “admonition” element. Markdown writer then turns this
into a fenced div with `title` class attribute. Since fenced divs
are block elements, their content is recognized as a paragraph
by the Markdown reader. This is an issue for Docbook writer because
it would produce an invalid DocBook document from such AST –
the `<title>` element can only contain “inline” elements.
Let’s handle this invalid special case separately by unwrapping
the paragraph before creating the `<title>` element.
Links with (internal) targets that the reader doesn't know about are
converted into emphasized text. Information on the link target is now
preserved by wrapping the text in a Span of class `spurious-link`, with
an attribute `target` set to the link's original target. This allows to
recover and fix broken or unknown links with filters.
See: #6916
Previously we used Setext (underlined) headings by default.
The default is now ATX (`##` style).
* Add the `--markdown-headings=atx|setext` option.
* Deprecate `--atx-headers`.
* Add constructor 'ATXHeadingInLHS` constructor to `LogMessage` [API change].
* Support `markdown-headings` in defaults files.
* Document new options in MANUAL.
Closes#6662.
For security reasons, some legal firms delete the date from comments and
tracked changes.
* Make date optional (Maybe) in tracked changes and comments datatypes
* Add tests
If the first element of a bulleted or ordered list is another list,
then that first item will disappear if the target format is docx. This
changes the docx writer so that it prepends an empty string for those
cases. With this, no items will disappear.
Closes#5948.
* Fix hlint suggestions, update hlint.yaml
Most suggestions were redundant brackets. Some required
LambdaCase.
The .hlint.yaml file had a small typo, and didn't ignore camelCase
suggestions in certain modules.
Writers.Tables is now Writers.AnnotatedTable. All of the types and
functions in it have had the "Ann" removed from them. Now it is
expected that the module be imported qualified.
Add Writers.Tables helper functions and types, add tests for those
The Writers.Tables module contains an AnnTable type that is a pandoc
Table with added inferred information that should be enough for
writers (in particular the HTML writer) to operate on without having
to lay out the table themselves.
The toAnnTable and fromAnnTable functions in that module convert
between AnnTable and Table. In addition to producing an AnnTable with
coherent and well-formed annotations, the toAnnTable function also
normalizes its input Table like the table builder does.
Various tests ensure that toAnnTable normalizes tables exactly like
the table builder, and that its annotations are coherent.
The lines of unknown keywords, like `#+SOMEWORD: value` are no longer
read as metadata, but kept as raw `org` blocks. This ensures that more
information is retained when round-tripping org-mode files;
additionally, this change makes it possible to support non-standard org
extensions via filters.
The behavior of the `#+AUTHOR` and `#+KEYWORD` export settings has
changed: Org now allows multiple such lines and adds a space between the
contents of each line. Pandoc now always parses these settings as meta
inlines; setting values are no longer treated as comma-separated lists.
Note that a Lua filter can be used to restore the previous behavior.
`#+DESCRIPTION` lines are treated as text with markup. If multiple such
lines are given, then all lines are read and separated by soft
linebreaks.
Closes: #6485
The `tex` export option can be set with `#+OPTION: tex:nil` and allows
three settings:
- `t` causes LaTeX fragments to be parsed as TeX or added as raw TeX,
- `nil` removes all LaTeX fragments from the document, and
- `verbatim` treats LaTeX as text.
The default is `t`.
Closes: #4070
The PandocError type is used throughout the Lua subsystem, all Lua
functions throw an exception of this type if an error occurs. The
`LuaException` type is removed and no longer exported from
`Text.Pandoc.Lua`. In its place, a new constructor `PandocLuaError` is
added to PandocError.
Now a cell with dimension (h, w) will be cut up into h*w cells of
dimension (1,1), all in the same grid position, with the upper-left
holding the original cell contents and the rest being empty.
The Builder.simpleTable now only adds a row to the TableHead when the
given header row is not null. This uncovered an inconsistency in the
readers: some would unconditionally emit a header filled with empty
cells, even if the header was not present. Now every reader has the
conditional behaviour. Only the XWiki writer depended on the header
row being always present; it now pads its head as necessary.
- Writers.Native is now adapted to the new Table type.
- Inline captions should now be conditionally wrapped in a Plain, not
a Para block.
- The toLegacyTable function now lives in Writers.Shared.
Jira text which is marked as `+inserted+` is converted into pandoc's
default representation for underlined text: a span with class
`underline`. Previously, the span was marked with the non-standard class
`inserted`.
Closes: #6237
Jira images attributes as in `!image.jpg|align=right!` are retained as
key-value pairs. Thumbnail images, such as `!example.gif|thumbnail!`,
are marked by a `thumbnail` class in their attributes.
Related to #6234.
* Simplify by collapsing a do block into a single <$>
* Remove an unnecessary variable: `all` takes any Foldable, so only blocksToInlines needs toList.
A bug was fixed which caused faulty parsing if a table was not preceded
by a newline and the first table cell had no space after the initial `|`
characters.
Fixes: #6198
A bug was fixed which caused non-emphasized text containing digits and/or
non-special symbols (like dots) to sometimes be parsed incorrectly.
Fixes: #6196
* Support for colored inlines has been added.
* Lists are now allowed to be indented; i.e., lists are still recognized
if list markers are preceded by spaces.
Closes: #6183, #6184
New formats:
- `jats_archiving` for the "Archiving and Interchange Tag Set",
- `jats_publishing` for the "Journal Publishing Tag Set", and
- `jats_articleauthoring` for the "Article Authoring Tag Set."
The "jats" output format is now an alias for "jats_archiving".
Closes: #6014
* Use concatMap instead of reimplementing it
* Replace an unnecessary multi-way if with a regular if
* Use sortOn instead of sortBy and comparing
* Use guards instead of lots of indents for if and else
* Remove redundant do blocks
* Extract common functions from both branches of maybe
Whenever both the Nothing and the Just branch of maybe do the same
function, do that function on the result of maybe instead.
* Use fmap instead of reimplementing it from maybe
* Use negative forms instead of negating the positive forms
* Use mapMaybe instead of mapping and then using catMaybes
* Use zipWith instead of mapping over the result of zip
* Use unwords instead of reimplementing it
* Use <$ instead of <$> and const
* Replace case of Bool with if and else
* Use find instead of listToMaybe and filter
* Use zipWithM instead of mapM and zip
* Inline lambda wrappers into the real functions
* We get zipWithM from Text.Pandoc.Writers.Shared
* Use maybe instead of fromMaybe and fmap
I'm not sure how this one slipped past me.
* Increase a bit of indentation
Lists of Inline and Block elements can now be filtered via `Inlines` and
`Blocks` functions, respectively. This is helpful if a filter conversion
depends on the order of elements rather than a single element.
For example, the following filter can be used to remove all spaces
before a citation:
function isSpaceBeforeCite (spc, cite)
return spc and spc.t == 'Space'
and cite and cite.t == 'Cite'
end
function Inlines (inlines)
for i = #inlines-1,1,-1 do
if isSpaceBeforeCite(inlines[i], inlines[i+1]) then
inlines:remove(i)
end
end
return inlines
end
Closes: #6038
The check whether a complex inline element following a string must be
escaped, now depends on the last character of the string instead of the
first.
Fixes: #5906
PR #5884.
+ Use pandoc-types 1.20 and texmath 0.12.
+ Text is now used instead of String, with a few exceptions.
+ In the MediaBag module, some of the types using Strings
were switched to use FilePath instead (not Text).
+ In the Parsing module, new parsers `manyChar`, `many1Char`,
`manyTillChar`, `many1TillChar`, `many1Till`, `manyUntil`,
`mantyUntilChar` have been added: these are like their
unsuffixed counterparts but pack some or all of their output.
+ `glob` in Text.Pandoc.Class still takes String since it seems
to be intended as an interface to Glob, which uses strings.
It seems to be used only once in the package, in the EPUB writer,
so that is not hard to change.
The first list item of a sublist should not resume numbering
from the number of the last sublist item of the same level,
if that sublist was a sublist of a different list item.
That is, we should not get:
```
1. one
1. sub one
2. sub two
2. two
3. sub one
```
* Set dbBook to true when traversing a chapter too.
Currently, a `<title/>` in a chapter and in a `<section/>` below that
chapter have the same level if they're not inside a `<book/>`.
This can happen in a multi-file book project. Also see the example at
https://tdg.docbook.org/tdg/4.5/chapter.html
Co-authored-by: Félix Baylac-Jacqué <felix@alternativebit.fr>
* Add docbook-chapter test
This tests nested `<section/>` and makes sure `<title/>` in the first
`<section/>` below `<chapter/>` is one level deeper than the `<chapter/>`'s
`<title/>`, also when not inside a `<book/>`.
Co-authored-by: Félix Baylac-Jacqué <felix@alternativebit.fr>
The new version of doctemplates adds many features to pandoc's
templating system, while remaining backwards-compatible.
New features include partials and filters. Using template filters,
one can lay out data in enumerated lists and tables.
Templates are now layout-sensitive: so, for example, if a
text with soft line breaks is interpolated near the end of
a line, the text will break and wrap naturally. This makes
the templating system much more suitable for programatically
generating markdown or other plain-text files from metadata.
Comment lines in Org-mode can be completely empty; both of these line
should produce no output:
# a comment
#
The reader used to produce a wrong result for the latter, but ignores
that line as well now.
Fixes: #5856
* HTML reader: Handle cite attribute for quotes. If a `<q>` tag has a `cite` attribute, we interpret it as a Quoted element with an inner Span. Closes#5798
* Refactor url canonicalization into a helper function
* Modify HTML writer to handle quote with cite.
[0]: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/q
Previously we used the following Project Gutenberg conventions
for plain output:
- extra space before and after level 1 and 2 headings
- all-caps for strong emphasis `LIKE THIS`
- underscores surrounding regular emphasis `_like this_`
This commit makes `plain` output plainer. Strong and Emph
inlines are rendered without special formatting. Headings
are also rendered without special formatting, and with only
one blank line following.
To restore the former behavior, use `-t plain+gutenberg`.
API change: Add `Ext_gutenberg` constructor to `Extension`.
See #5741.
Motivating issues: #5523, #5052, #5074
Style name comparisons are case-insensitive, since those are
case-insensitive in Word.
w:styleId will be used as style name if w:name is missing (this should
only happen for malformed docx and is kept as a fallback to avoid
failing altogether on malformed documents)
Block quote detection code moved from Docx.Parser to Readers.Docx
Code styles, i.e. "Source Code" and "Verbatim Char" now honor style
inheritance
Docx Reader now honours "Compact" style (used in Pandoc-generated docx).
The side-effect is that "Compact" style no longer shows up in
docx+styles output. Styles inherited from "Compact" will still
show up.
Removed obsolete list-item style from divsToKeep. That didn't
really do anything for a while now.
Add newtypes to differentiate between style names, ids, and
different style types (that is, paragraph and character styles)
Since docx style names can have spaces in them, and pandoc-markdown
classes can't, anywhere when style name is used as a class name,
spaces are replaced with ASCII dashes `-`.
Get rid of extraneous intermediate types, carrying styleId information.
Instead, styleId is saved with other style data.
Use RunStyle for inline style definitions only (lacking styleId and styleName);
for Character Styles use CharStyle type (which is basicaly RunStyle with styleId
and StyleName bolted onto it).
This commit prevents custom styles on divs and spans from overriding
styles on certain elements inside them, like headings, blockquotes,
and links. On those elements, the "native" style is required for the
element to display correctly. This change also allows nesting of
custom styles; in order to do so, it removes the default "Compact"
style applied to Plain blocks, except when inside a table.
* Require recent doctemplates. It is more flexible and
supports partials.
* Changed type of writerTemplate to Maybe Template instead
of Maybe String.
* Remove code from the LaTeX, Docbook, and JATS writers that looked in
the template for strings to determine whether it is a book or an
article, or whether csquotes is used. This was always kludgy and
unreliable. To use csquotes for LaTeX, set `csquotes` in your
variables or metadata. It is no longer sufficient to put
`\usepackage{csquotes}` in your template or header includes.
To specify a book style, use the `documentclass` variable or
`--top-level-division`.
* Change template code to use new API for doctemplates.
The `raw_attribute` will be used to mark raw bits, even HTML
and LaTeX, and even when `raw_html` and `raw_tex` are enabled,
as they are by default.
To get the old behavior, disable `raw_attribute` in the writer.
Closes#4311.
This commit alters the way in which the Powerpoint writer treats
inline code and code blocks.
- Inline code is now formatted at the same size as the surrounding
text.
- Code blocks are now given a margin and font size according to their
level.
- Furthermore this commit allows changing the font with which code is
formatted via the `monofont` option.
Tested in
- PowerPoint 365 for Windows - 1808 (Build 10730.20344 Click-to-Run)
- PowerPoint 365 for Mac - 16.26 (19060901)
Version specifiers like `PANDOC_VERSION` and `PANDOC_API_VERSION` are
turned into `Version` objects. The objects simplify version-appropriate
comparisons while maintaining backward-compatibility.
A function `pandoc.types.Version` is added as part of the newly
introduced module `pandoc.types`, allowing users to create version
objects in scripts.
This makes use of tasty-lua, a package to write tests in Lua
and integrate the results into Tasty output. Test output becomes
more informative: individual tests and test groups become visible
in test output. Failures are reported with helpful error messages.
"Best Practice: When footnotes are grouped at the end of an article,
wrap them in a `<fn-group>` and use an `<xref>` element in the text, as
usual, to tie each footnote in the list to a particular location in the
text."
Closes#5511.
Planning info is now always placed before the subtree contents.
Previously, the planning info was placed after the content if the
header's subtree was converted to a list, which happens with headers of
level 3 and higher per default.
Fixes: #5494
The `system` Lua module provides utility functions to interact with the
operating- and file system. E.g.
print(pandoc.system.get_current_directory())
or
pandoc.system.with_temporary_directory('tikz', function (dir)
-- write and compile a TikZ file with pdflatex
end)
