mbsync: per account multiple channels (#1360)

* mbsync: option for configuring a channel

A channel is a relationship between 2 directories/boxes/mailboxes
between the local machine (slave) and the remote mail server (master).
Each channel must be given at least:
     * an account-unique name
     * a pattern for which mailboxes to sync from master
     * a pattern for what directory where that mail ends up on the
     slave

Additional options can be added later.

* mbsync: option for configuring a group

A group is a grouping of channels together, so that many channels with
very different names can be handled as a single entity.

Groups are unique in mbsync because they will shadow channels that
have the same name on the command-line.

* mbsync: create groups configuration attribute

This is the end of the configuration that the end-user will use.

They will specify an attribute set that contains the name for the
group, so they can say
`accounts.email.accounts.<aname>.groups.<gname>` to access the
configuration for the group with the name `<gname>`.

* mbsync: write function to generate group-channel blocks

This function takes in a set of groups, and their consituent
channels and writes the appropriate .mbsyncrc block. The block is as
shown below:

      Group groupName1
      Channel channelName1
      Channel channelName2

      Group groupName2
      Channel channelName3

Each group must have a unique name, no matter which account it is
declared under. The same holds true for channels. However, if there is
a group that shares the same name as the channel, the channel will
effectively be "shadowed" by the group, and mbsync will default to
working with the group in that case.

* mbsync: write function to generate channel configuration blocks

This function takes in a set of groups, which includes their
consituent channels and writes the appropriate .mbsyncrc block for the
channel. The block that is generated is shown below:
      Channel groupName1-channelName1
      Master :<accountName>-remote:<master-pattern>
      Slave :<accountName>-local:<slave-pattern>

      Channel groupName2-channelName2
      Master :<accountName>-remote:<master-pattern>
      Slave :<accountName>-local:<slave-pattern>

Each group must have a unique name, no matter which account it is
declared under. The same holds true for channels.

Using channels with the patterns set up this way allows one to specify
which maildir directories are to be synchronized FROM the master TO
the slave. In addition, it allows for these maildirs to be remapped,
between the master server and the local slave.
This is critical, because Gmail has a strange way of storing its mail
that makes using mbsync, mu, and mu4e more difficult.

There are additional channel parameters that are already present in
this codebase from the previous use of group-channel configuration,
which will be reused.

* mbsync: set the submodule's names field according to parameter

This is the same method as is used in creating an email account, named
`<name>` under `accounts.email.accounts.<name>`. This allows the user
to specify groups and channels, in a list-like format, but still gets
the "namespacing" to more easily handle the options available in each
of these locations.

* mbsync: provide examples of master/slave patterns for channels

* mbsync: create nested-let function to generate channel pattern

This pattern is required to either NOT be present, which means the
master pattern is used to match, or it has a list of patterns to use
beneath the master maildir to match against.

This function checks to ensure that if patterns is not empty, ONLY
then is the `Pattern` keyword printed. Otherwise, there are many, many
problems.
If there IS a list of patterns, then we use proper escaping methods to
ensure that the exact string is constructed.

* mbsync: per-account groups can have additional patterns

Gave the
`accounts.email.accounts.<name>.mbsync.groups.<gname>.channel.<cname>`
set a `patterns` option, which will allow for greater customization
and filtering of the master maildir to sync to the slave maildir.

* mbsync: add extraConfig option for easier-to-format options

These are options that can be handled by the `genSection` function in
the `genAccountFunction`, so they are left to the user to decide.
Most of these are made on a global basis anyways.

* mbsync: remove unneeded extraConfig.channel

This was originally placed here, seemingly, just to get this module
working. However, this field is actually more confusing now that a
separate per-channel configuration option for extra configurations has
been made available.

* mbsync: correct and improve comment in masterPattern description

* mbsync: switch channel/group generation to new functions

Changing this out is what moves us from the old system to the new one.
Instead of having a single channel manage a whole mailbox, we can now
specify an attribute set of groups that should correspond to an email
account.

Each of these groups contains an attribute set of channels that make
it up, and are grouped together for synchronization. In addition, each
of these channels can have additional IMAP4 parameters attached to
them to further refine synchronization.

Lastly, each of the channels is grouped together under the Group
section, ensuring that the channels' mailboxes synchronize as they
have been specified.

* mbsync: only generate group/channel configuration if channels present

Typically, when a group is specified, channels will be specified as
well. However, if due to error or mistake, the user forgets to specify
ANY channels for a group, we should not generate that group's
information.

This means that no channels are specified (which maps the remote
master to local slave). In addition, the `Group <gName>` block (which
brings the separate channels together) is also not generated.

Another thing to consider is that a user might specify a group and a
channel, but perform no additional configuration of the channel.
In a configuration, this would be realized by
`accounts.email.accounts.<aName>.mbsync.groups.<gName>.channels.<cName>;`

This creates the channel with the name `<cName>` and the
`masterPattern`, `slavePattern`, and `patterns` fields use their defaults.
By definitions set within mbsync, these defaults actually specify that
the remote master's `INBOX` mail directory is synchronized to the
local slave's `INBOX` directory.

So, if there is a channel that has no fields specified, then we DO
want to generate its configuration. But if there is a group that has
no channels, then we do NOT generate it.

* mbsync: acc comment explaining why groups attr set is never empty

* Revert "mbsync: remove unneeded extraConfig.channel"

This reverts commit 941c4771ca.

To support backwards compatibility, I need to leave this field/option
in the module, even if it will likely be more confusing to do it this way.

* mbsync: channel compatibility with previous iteration of mbsync

The previous version of mbsync used a single channel for an entire
account. This leads to issues when trying to change the mailbox
hierarchy on the local machine. The problem with this is that some
email providers (Gmail, among others) use a slightly different maildir
hierarchy, where the standard mailboxes (Inbox, Drafts, Trash, etc.)
are stored inside another directory (`[Gmail]/` in the case of Gmail).

This new version allows the user to specify any number of groups with
any number of channels within to reorder their mail however they wish.

However, to maintain backwards compatibility, I moved the original
channel-generating code to a function that will run ONLY when
there are no groups specified for THIS account.

* Revert "mbsync: channel compatibility with previous iteration of mbsync"

This reverts commit b1a241ff9f.

This function is in the wrong location and this was wrongly committed.

* mbsync: function for backwards compatibility with previous mbsync

NOTE THAT THIS IS THE CORRECT COMMIT FOR THIS CHUNK OF CODE!!

The previous version of mbsync used a single channel for an entire
account. This leads to issues when trying to change the mailbox
hierarchy on the local machine. The problem with this is that some
email providers (Gmail, among others) use a slightly different maildir
hierarchy, where the standard mailboxes (Inbox, Drafts, Trash, etc.)
are stored inside another directory (`[Gmail]/` in the case of Gmail).

This new version allows the user to specify any number of groups with
any number of channels within to reorder their mail however they wish.

However, to maintain backwards compatibility, I moved the original
channel-generating code to a function that will run ONLY when
there are no groups specified for THIS account.

* mbsync: function to choose which style of group/channels to generate

This is a simple if-check. If the old style is used, then this
account's mbsync.groups attribute set is empty. If that is the case,
then the old-style single-channel per account is used.

If that is NOT the case, then the new style is used in preference of
the old. This means that ALL channel code that would be generated by
the old version is replaced by the new one.

* mbsync: switch per-account config generation to check channels

* mbsync: program-wide groups if no account-specific groups

At the end, we have to choose whether or not to generate the old style
of having program-wide groups to specify things, where the boxes on
the channel underneath the group specifies which mailboxes to sync.

Here, we only generate the old style of group IF there is ANY account
that does NOT have the new `accounts.mbsync.groups` defined. At that
point, it is up to the user to ensure that the accounts in
`programs.mbsync.groups.{}` align with the name chosen for the
account, as I have made no attempt to change this old code.

However, if ALL accounts have their `mbsync.groups` defined, even if
each of the groups has a single empty channel, it will generate the
groups in the new style.

* mbsync: ensure \n after hm-generated comment

This was a multi-part fix. First, the `# Generated by Home Manager.`
comment has been reworked to ensure that it will ALWAYS have a
newline, even if the program-wide extraConfiguration is empty.

Next, we switched to placing 2 newlines between every account, to
provide further visual distinction between each account, which can
have multiple channels and multiple groups defined at the same time.

Lastly, the groupsConfig was slightly reworked, so that both the old
and new version can be used, but the new one will take precedence.
Because of this, groupsConfig is now a list of strings, which will
have single newlines inserted between each element.

But if the old style is NOT used, then the groupsConfig list
contains one element, an empty string. A single element has nothing
added as a separator, and an empty string produces no output.

* mbsync: only generate new group/channels if channels present

Here, the problem was if the user created a group for an account, but
did not also include a set of channels. If no channels have been
specified, then the group should NOT have its group-channel mapping generated.

I also corrected and improved the comment regarding
`genGroupChannelString`'s function and intended behavior.

* mbsync: channel patterns generate their own newlines

This means that when a channel has extra `patterns` defined for it, it
will generate those, and a single newline will be appended to the end
of that newly constructed string.

The moving of the newline character is slightly important because
otherwise, every account would receive an extra newline after every
channel, leading to 2 newlines after every channel.

* mbsync: place newline between each channel in a group

* mbsync: ensure old group/channel has proper spacing

This ensures that if the old style of generating program-wide groups
that there is the proper spacing before the group and in between each
line within the group.

* mbsync: ensure no empty channels present

If the user specifies a group correctly, they must still specify an
attribute set of channels. However, if they do not, then we need to
ensure that a group with no channels does NOT have any channel
configurations generated for it.

If there is a channel string generated for a channel that is empty,
then the `mapAttrsToList` returns a singleton list that contains just
the empty string. Thus, we can filter out all those results, to ensure
that no empty channels are generated.

It is important to keep in mind the difference between an empty
channel and a channel that has received no configuration, but is
named.
	* A named channel is technically configured to have a name.
	  While the `masterPattern`, `slavePattern`, and `patterns`
	  field have NOT been populated, mbsync assumes that if
	  master/slave-Pattern are empty that means match against
	  `INBOX`.
	  If `patterns` is empty, no patterns are printed.
	* An empty channel set is a set that has no channels within
	  it, but `mbsync.groups.<gName>.channels` is defined.

* mbsync: filter empty groups and correct newlines

First thing, someone can specify that a group is empty. If this is
done, technically a group with channels would be generated at the end.
However, because they were empty and did not exist, whitespacing would
be generated, leading to a usable, but mangled config file.
The `filter` solves this problem by removing empty strings (which are
generated by groups that are empty) from the output strings to place
in the file.

Lastly, because the whitespacing was fixed elsewhere in the file, the
crazy double-newline at the end was changed to a single newline.
However, the double newline within the `concatStringsSep` is still
required, because the list that is being concatenated together is a
list of channel configurations. Each element corresponds to one of the
groups specified, whose contents are the channels specified within.

The double newline is needed because each string element is lacking a
trailing newline, because `concatStringsSep` does not add the
separator to the end of the last element in the list. So, the last
channel to be configured will not have that newline appended when the
channel-configuration list is created, thus, 2 are inserted here.

* mbsync: update test input to use per-account channels

* mbsync: comment how old/new style collision handled

This is left in the test input for now, because I think it is useful
to see why certain things are happening the way they are.

* mbsync: update test output pattern

The test output should now have the correct configuration according to
the way I have specified it in the input file.

* mbsync: use format script on new code

* mbsync: add KarlJoad as maintainer

Co-authored-by: Nick Hu <me@nickhu.co.uk>
This commit is contained in:
Karl H 2020-09-21 12:16:06 -05:00 committed by GitHub
parent 9b1b55ba02
commit 96d7de6db1
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
5 changed files with 321 additions and 33 deletions

3
.github/CODEOWNERS vendored
View File

@ -76,6 +76,9 @@
/modules/programs/matplotlib.nix @rprospero
/modules/programs/mbsync.nix @KarlJoad
/tests/modules/programs/mbsync @KarlJoad
/modules/programs/mcfly.nix @marsam
/modules/programs/mpv.nix @tadeokondrak

View File

@ -1,4 +1,4 @@
{ lib, ... }:
{ config, lib, ... }:
with lib;
@ -6,6 +6,110 @@ let
extraConfigType = with lib.types; attrsOf (either (either str int) bool);
perAccountGroups = { name, config, ... }: {
options = {
name = mkOption {
type = types.str;
# Make value of name the same as the name used with the dot prefix
default = name;
readOnly = true;
description = ''
The name of this group for this account. These names are different than
some others, because they will hide channel names that are the same.
'';
};
channels = mkOption {
type = types.attrsOf (types.submodule channel);
default = { };
description = ''
List of channels that should be grouped together into this group. When
performing a synchronization, the groups are synchronized, rather than
the individual channels.
</para><para>
Using these channels and then grouping them together allows for you to
define the maildir hierarchy as you see fit.
'';
};
};
};
# Options for configuring channel(s) that will be composed together into a group.
channel = { name, config, ... }: {
options = {
name = mkOption {
type = types.str;
default = name;
readOnly = true;
description = ''
The unique name for THIS channel in THIS group. The group will refer to
this channel by this name.
</para><para>
In addition, you can manually sync just this channel by specifying this
name to mbsync on the command line.
'';
};
masterPattern = mkOption {
type = types.str;
default = "";
example = "[Gmail]/Sent Mail";
description = ''
IMAP4 patterns for which mailboxes on the remote mail server to sync.
If <literal>Patterns</literal> are specified, <literal>masterPattern</literal>
is interpreted as a prefix which is not matched against the patterns,
and is not affected by mailbox list overrides.
</para><para>
If this is left as the default, then mbsync will default to the pattern
<literal>INBOX</literal>.
'';
};
slavePattern = mkOption {
type = types.str;
default = "";
example = "Sent";
description = ''
Name for where mail coming from the master mail server will end up
locally. The mailbox specified by the master's pattern will be placed
in this directory.
</para><para>
If this is left as the default, then mbsync will default to the pattern
<literal>INBOX</literal>.
'';
};
patterns = mkOption {
type = types.listOf types.str;
default = [ ];
example = [ "INBOX" ];
description = ''
Instead of synchronizing <emphasis>just</emphasis> the mailboxes that
match the <literal>masterPattern</literal>, use it as a prefix which is
not matched against the patterns, and is not affected by mailbox list
overrides.
'';
};
extraConfig = mkOption {
type = extraConfigType;
default = { };
example = ''
{
Create = "both";
CopyArrivalDate = "yes";
MaxMessages = 10000;
MaxSize = "1m";
}
'';
description = ''
Extra configuration lines to add to <emphasis>THIS</emphasis> channel's
configuration.
'';
};
};
};
in {
options.mbsync = {
enable = mkEnableOption "synchronization using mbsync";
@ -58,6 +162,23 @@ in {
'';
};
groups = mkOption {
type = types.attrsOf (types.submodule perAccountGroups);
default = { };
# The default cannot actually be empty, but contains an attribute set where
# the channels set is empty. If a group is specified, then a name is given,
# creating the attribute set.
description = ''
Some email providers (Gmail) have a different directory hierarchy for
synchronized email messages. Namely, when using mbsync without specifying
a set of channels into a group, all synchronized directories end up beneath
the <literal>[Gmail]/</literal> directory.
</para><para>
This option allows you to specify a group, and subsequently channels that
will allow you to sync your mail into an arbitrary hierarchy.
'';
};
extraConfig.channel = mkOption {
type = extraConfigType;
default = { };

View File

@ -69,23 +69,98 @@ let
Inbox = "${maildir.absPath}/${folders.inbox}";
SubFolders = "Verbatim";
} // optionalAttrs (mbsync.flatten != null) { Flatten = mbsync.flatten; }
// mbsync.extraConfig.local) + "\n" + genSection "Channel ${name}" ({
Master = ":${name}-remote:";
Slave = ":${name}-local:";
Patterns = mbsync.patterns;
Create = masterSlaveMapping.${mbsync.create};
Remove = masterSlaveMapping.${mbsync.remove};
Expunge = masterSlaveMapping.${mbsync.expunge};
SyncState = "*";
} // mbsync.extraConfig.channel) + "\n";
// mbsync.extraConfig.local) + "\n" + genChannels account;
genChannels = account:
with account;
if mbsync.groups == { } then
genAccountWideChannel account
else
genGroupChannelConfig name mbsync.groups + "\n"
+ genAccountGroups mbsync.groups;
# Used when no channels are specified for this account. This will create a
# single channel for the entire account that is then further refined within
# the Group for synchronization.
genAccountWideChannel = account:
with account;
genSection "Channel ${name}" ({
Master = ":${name}-remote:";
Slave = ":${name}-local:";
Patterns = mbsync.patterns;
Create = masterSlaveMapping.${mbsync.create};
Remove = masterSlaveMapping.${mbsync.remove};
Expunge = masterSlaveMapping.${mbsync.expunge};
SyncState = "*";
} // mbsync.extraConfig.channel) + "\n";
# Given the attr set of groups, return a string of channels that will direct
# mail to the proper directories, according to the pattern used in channel's
# master pattern definition.
genGroupChannelConfig = storeName: groups:
let
# Given the name of the group this channel is part of and the channel
# itself, generate the string for the desired configuration.
genChannelString = groupName: channel:
let
escapeValue = escape [ ''\"'' ];
hasSpace = v: builtins.match ".* .*" v != null;
# Given a list of patterns, will return the string requested.
# Only prints if the pattern is NOT the empty list, the default.
genChannelPatterns = patterns:
if (length patterns) != 0 then
"Pattern " + concatStringsSep " "
(map (pat: if hasSpace pat then escapeValue pat else pat)
patterns) + "\n"
else
"";
in genSection "Channel ${groupName}-${channel.name}" ({
Master = ":${storeName}-remote:${channel.masterPattern}";
Slave = ":${storeName}-local:${channel.slavePattern}";
} // channel.extraConfig) + genChannelPatterns channel.patterns;
# Given the group name, and a attr set of channels within that group,
# Generate a list of strings for each channels' configuration.
genChannelStrings = groupName: channels:
optionals (channels != { })
(mapAttrsToList (channelName: info: genChannelString groupName info)
channels);
# Given a group, return a string that configures all the channels within
# the group.
genGroupsChannels = group:
concatStringsSep "\n" (genChannelStrings group.name group.channels);
# Generate all channel configurations for all groups for this account.
in concatStringsSep "\n" (filter (s: s != "")
(mapAttrsToList (name: group: genGroupsChannels group) groups));
# Given the attr set of groups, return a string which maps channels to groups
genAccountGroups = groups:
let
# Given the name of the group and the attribute set of channels, make
# make "Channel <grpName>-<chnName>" for each channel to list os strings
genChannelStrings = groupName: channels:
mapAttrsToList (name: info: "Channel ${groupName}-${name}") channels;
# Take in 1 group, if the group has channels specified, construct the
# "Group <grpName>" header and each of the channels.
genGroupChannelString = group:
flatten (optionals (group.channels != { }) ([ "Group ${group.name}" ]
++ (genChannelStrings group.name group.channels)));
# Given set of groups, generates list of strings, where each string is one
# of the groups and its consituent channels.
genGroupsStrings = mapAttrsToList (name: info:
concatStringsSep "\n" (genGroupChannelString groups.${name})) groups;
in concatStringsSep "\n\n" (filter (s: s != "")
genGroupsStrings) # filter for the cases of empty groups
+ "\n"; # Put all strings together.
genGroupConfig = name: channels:
let
genGroupChannel = n: boxes: "Channel ${n}:${concatStringsSep "," boxes}";
in concatStringsSep "\n"
in "\n" + concatStringsSep "\n"
([ "Group ${name}" ] ++ mapAttrsToList genGroupChannel channels);
in {
meta.maintainers = [ maintainers.KarlJoad ];
options = {
programs.mbsync = {
enable = mkEnableOption "mbsync IMAP4 and Maildir mailbox synchronizer";
@ -150,11 +225,20 @@ in {
home.file.".mbsyncrc".text = let
accountsConfig = map genAccountConfig mbsyncAccounts;
groupsConfig = mapAttrsToList genGroupConfig cfg.groups;
in concatStringsSep "\n" ([''
# Only generate this kind of Group configuration if there are ANY accounts
# that do NOT have a per-account groups/channels option(s) specified.
groupsConfig =
if any (account: account.mbsync.groups == { }) mbsyncAccounts then
mapAttrsToList genGroupConfig cfg.groups
else
[ ];
in ''
# Generated by Home Manager.
''] ++ optional (cfg.extraConfig != "") cfg.extraConfig ++ accountsConfig
++ groupsConfig) + "\n";
''
+ concatStringsSep "\n" (optional (cfg.extraConfig != "") cfg.extraConfig)
+ concatStringsSep "\n\n" accountsConfig
+ concatStringsSep "\n" groupsConfig;
home.activation = mkIf (mbsyncAccounts != [ ]) {
createMaildir =

View File

@ -15,14 +15,41 @@ Inbox /home/hm-user/Mail/hm-account/Inbox
Path /home/hm-user/Mail/hm-account/
SubFolders Verbatim
Channel hm-account
Create None
Expunge None
Channel emptyChannels-empty1
Master :hm-account-remote:
Patterns *
Remove None
Slave :hm-account-local:
SyncState *
Channel emptyChannels-empty2
Master :hm-account-remote:
Slave :hm-account-local:
Channel hm-account-earlierPatternMatch
Master :hm-account-remote:Label
Slave :hm-account-local:SomethingUnderLabel
Pattern ThingUnderLabel !NotThisMaildirThough "[Weird] Label?"
Channel hm-account-inbox
Master :hm-account-remote:Inbox
Slave :hm-account-local:Inbox
Channel hm-account-patternMatch
Master :hm-account-remote:Label
Slave :hm-account-local:SomethingUnderLabel
Pattern ThingUnderLabel !NotThisMaildirThough "[Weird] Label?"
Channel hm-account-strangeHostBoxName
Master ":hm-account-remote:[Weird]/Label Mess"
Slave :hm-account-local:[AnotherWeird]/Label
Group emptyChannels
Channel emptyChannels-empty1
Channel emptyChannels-empty2
Group hm-account
Channel hm-account-earlierPatternMatch
Channel hm-account-inbox
Channel hm-account-patternMatch
Channel hm-account-strangeHostBoxName
IMAPAccount hm@example.com
@ -40,16 +67,14 @@ Inbox /home/hm-user/Mail/hm@example.com/Inbox
Path /home/hm-user/Mail/hm@example.com/
SubFolders Verbatim
Channel hm@example.com
Create None
Expunge None
Master :hm@example.com-remote:
Patterns *
Remove None
Slave :hm@example.com-local:
SyncState *
Channel inboxes-inbox1
Master :hm@example.com-remote:Inbox1
Slave :hm@example.com-local:Inboxes
Channel inboxes-inbox2
Master :hm@example.com-remote:Inbox2
Slave :hm@example.com-local:Inboxes
Group inboxes
Channel hm-account:Inbox
Channel hm@example.com:Inbox1,Inbox2
Channel inboxes-inbox1
Channel inboxes-inbox2

View File

@ -8,6 +8,10 @@ with lib;
config = {
programs.mbsync = {
enable = true;
# programs.mbsync.groups and
# accounts.email.accounts.<name>.mbsync.groups should NOT be used at the
# same time.
# If they are, then the new version will take precendence.
groups.inboxes = {
"hm@example.com" = [ "Inbox1" "Inbox2" ];
hm-account = [ "Inbox" ];
@ -15,9 +19,60 @@ with lib;
};
accounts.email.accounts = {
"hm@example.com".mbsync = { enable = true; };
"hm@example.com".mbsync = {
enable = true;
groups.inboxes = {
channels = {
inbox1 = {
masterPattern = "Inbox1";
slavePattern = "Inboxes";
};
inbox2 = {
masterPattern = "Inbox2";
slavePattern = "Inboxes";
};
};
};
};
hm-account.mbsync = { enable = true; };
hm-account.mbsync = {
enable = true;
groups.hm-account = {
channels.earlierPatternMatch = {
masterPattern = "Label";
slavePattern = "SomethingUnderLabel";
patterns = [
"ThingUnderLabel"
"!NotThisMaildirThough"
''"[Weird] Label?"''
];
};
channels.inbox = {
masterPattern = "Inbox";
slavePattern = "Inbox";
};
channels.strangeHostBoxName = {
masterPattern = "[Weird]/Label Mess";
slavePattern = "[AnotherWeird]/Label";
};
channels.patternMatch = {
masterPattern = "Label";
slavePattern = "SomethingUnderLabel";
patterns = [
"ThingUnderLabel"
"!NotThisMaildirThough"
''"[Weird] Label?"''
];
};
};
# No group should be printed.
groups.emptyGroup = { };
# Group should be printed, but left with default channels.
groups.emptyChannels = {
channels.empty1 = { };
channels.empty2 = { };
};
};
};
nmt.script = ''