pandoc/test/command/4742.md

Check that the commonmark reader handles the `ascii_identifiers`
extension properly.

```
% pandoc -f commonmark+gfm_auto_identifiers+ascii_identifiers -t native
# non ascii ⚠️ räksmörgås
^D
[Header 1 ("non-ascii--raksmorgas",[],[]) [Str "non",Space,Str "ascii",Space,Str "\9888\65039",Space,Str "r\228ksm\246rg\229s"]]
```

Note that the emoji here is actually a composite character,
formed from \9888 and \65039. The latter is a combining mark,
so it survives...

```
% pandoc -f commonmark+gfm_auto_identifiers-ascii_identifiers -t native
# non ascii ⚠️ räksmörgås
^D
[Header 1 ("non-ascii-\65039-r\228ksm\246rg\229s",[],[]) [Str "non",Space,Str "ascii",Space,Str "\9888\65039",Space,Str "r\228ksm\246rg\229s"]]
```

`gfm` should have `ascii_identifiers` disabled by default.

```
% pandoc -f gfm -t native
# non ascii ⚠️ räksmörgås
^D
[Header 1 ("non-ascii-\65039-r\228ksm\246rg\229s",[],[]) [Str "non",Space,Str "ascii",Space,Str "\9888\65039",Space,Str "r\228ksm\246rg\229s"]]
```
CommonMark reader: Handle ascii_identifiers extension (#4733) Non-ascii characters were not stripped from identifiers even if the `ascii_identifiers` extension was enabled (which is is by default for gfm). Closes #4742 2018-06-29 10:41:26 +02:00			Check that the commonmark reader handles the `ascii_identifiers`
			`extension properly.`

			```
`commonmark_x`: replace `auto_identifiers` with `gfm_auto_identifiers`. `commonmark_x` never actually supported `auto_identifiers` (it didn't do anything), because the underlying library implements gfm-style identifiers only. Attempts to add the `autolink_identifiers` extension to `commonmark` will now fail with an error. Closes #6863. 2020-11-20 18:17:14 +01:00			`% pandoc -f commonmark+gfm_auto_identifiers+ascii_identifiers -t native`
CommonMark reader: Handle ascii_identifiers extension (#4733) Non-ascii characters were not stripped from identifiers even if the `ascii_identifiers` extension was enabled (which is is by default for gfm). Closes #4742 2018-06-29 10:41:26 +02:00			`# non ascii ⚠️ räksmörgås`
			`^D`
			`[Header 1 ("non-ascii--raksmorgas",[],[]) [Str "non",Space,Str "ascii",Space,Str "\9888\65039",Space,Str "r\228ksm\246rg\229s"]]`
			```

Exactly match GitHub's identifier generating algorithm. See #5057. 2018-11-12 05:45:38 +01:00			`Note that the emoji here is actually a composite character,`
			`formed from \9888 and \65039. The latter is a combining mark,`
			`so it survives...`

CommonMark reader: Handle ascii_identifiers extension (#4733) Non-ascii characters were not stripped from identifiers even if the `ascii_identifiers` extension was enabled (which is is by default for gfm). Closes #4742 2018-06-29 10:41:26 +02:00			```
`commonmark_x`: replace `auto_identifiers` with `gfm_auto_identifiers`. `commonmark_x` never actually supported `auto_identifiers` (it didn't do anything), because the underlying library implements gfm-style identifiers only. Attempts to add the `autolink_identifiers` extension to `commonmark` will now fail with an error. Closes #6863. 2020-11-20 18:17:14 +01:00			`% pandoc -f commonmark+gfm_auto_identifiers-ascii_identifiers -t native`
CommonMark reader: Handle ascii_identifiers extension (#4733) Non-ascii characters were not stripped from identifiers even if the `ascii_identifiers` extension was enabled (which is is by default for gfm). Closes #4742 2018-06-29 10:41:26 +02:00			`# non ascii ⚠️ räksmörgås`
			`^D`
			`[Header 1 ("non-ascii-\65039-r\228ksm\246rg\229s",[],[]) [Str "non",Space,Str "ascii",Space,Str "\9888\65039",Space,Str "r\228ksm\246rg\229s"]]`
			```

Text.Pandoc.Shared: add parameter to uniqueIdent, inlineListToIdentifier. The parameter is Extensions. This allows these functions to be sensitive to the settings of `Ext_gfm_auto_identifiers` and `Ext_ascii_identifiers`. This allows us to use `uniqueIdent` in the CommonMark reader, replacing some custom code. It also means that `gfm_auto_identifiers` can now be used in all formats. Semantically, `gfm_auto_identifiers` is now a modifier of `auto_identifiers`; for identifiers to be set, `auto_identifiers` must be turned on, and then the type of identifier produced depends on `gfm_auto_identifiers` and `ascii_identifiers` are set. Closes #5057. 2018-11-11 22:27:25 +01:00			`gfm` should have `ascii_identifiers` disabled by default.
CommonMark reader: Handle ascii_identifiers extension (#4733) Non-ascii characters were not stripped from identifiers even if the `ascii_identifiers` extension was enabled (which is is by default for gfm). Closes #4742 2018-06-29 10:41:26 +02:00
			```
			`% pandoc -f gfm -t native`
			`# non ascii ⚠️ räksmörgås`
			`^D`
Text.Pandoc.Shared: add parameter to uniqueIdent, inlineListToIdentifier. The parameter is Extensions. This allows these functions to be sensitive to the settings of `Ext_gfm_auto_identifiers` and `Ext_ascii_identifiers`. This allows us to use `uniqueIdent` in the CommonMark reader, replacing some custom code. It also means that `gfm_auto_identifiers` can now be used in all formats. Semantically, `gfm_auto_identifiers` is now a modifier of `auto_identifiers`; for identifiers to be set, `auto_identifiers` must be turned on, and then the type of identifier produced depends on `gfm_auto_identifiers` and `ascii_identifiers` are set. Closes #5057. 2018-11-11 22:27:25 +01:00			`[Header 1 ("non-ascii-\65039-r\228ksm\246rg\229s",[],[]) [Str "non",Space,Str "ascii",Space,Str "\9888\65039",Space,Str "r\228ksm\246rg\229s"]]`
CommonMark reader: Handle ascii_identifiers extension (#4733) Non-ascii characters were not stripped from identifiers even if the `ascii_identifiers` extension was enabled (which is is by default for gfm). Closes #4742 2018-06-29 10:41:26 +02:00			```