1.1 Introduction

The popstore is primarily designed for scalability. Central database files, a principal cause of bottlenecks in high volume settings, are avoided.¹ In a similar vein, the underlying message store itself can be spread across any number of disks.

The major components of the popstore are described below.

Legacy and popstore POP3 server

A dual-store, multi-threaded POP3 server is provided on UNIX and OpenVMS platforms which supports both the PMDF popstore and the legacy UNIX Berkeley and OpenVMS MAIL mailbox formats.

MessageStore and popstore POP3 server

A new dual-store, multi-threaded POP server is provided on all platforms which supports both the popstore and MessageStore. This server includes security and performance enhancements not possible while maintaining support for legacy mailbox formats.

Poppassd server

A multi-threaded poppassd server for users of Mulberry, Eudora, and other clients which support the ad hoc poppassd protocol for changing passwords.

Web-based user interface

A basic web-based user interface is provided. This interface allows users to use a web-client to change their password as well as see basic usage information about their popstore account. They can also read and delete messages stored for their account. For details, see Chapter 5.

Web-based management utility

A web-based management utility to manage the popstore. The utility presents itself as a multi-threaded CGI accessed through the PMDF HTTP server. Popstore users with management privileges can use this interface to monitor and manage the popstore. This utility is extremely reconfigurable; the entire interface can be changed around to suit a site's needs. See Chapter 4 for a description of this interface.

Command line management utility

A command line oriented management utility. Users with operating system privileges as well as popstore users who have been granted popstore management privileges can use the utility. See Chapters 6 or 7 for information about this utility.

Migration utility

A utility is provided to migrate the mail inboxes for login accounts to the popstore. The utility can create a popstore account for each migrated user, migrate their mail inbox, and then establish mail forwarding from their login account's message store to the popstore. See Chapter 8 for details about this utility.

Forwarding database

A forwarding database which allows mail for popstore users, fictitious or otherwise, to be automatically redirected elsewhere. Consult Section 1.5 for further details.

Delivery channel

A master channel to deliver inbound messages to the popstore. Inbound messages for the popstore are queued to this channel by PMDF. The channel is then run by PMDF to deliver the messages to the popstore. See Section 10.1 for details.

Message bouncer

This is a job which runs periodically and either returns or deletes old stored messages which have "expired". This channel is best likened to the PMDF RETURN job. This job is used to "time out" old messages which popstore users have not deleted. If an old message has never been read, it is returned as undelivered. If it has been read, it is deleted. Note that the popstore can be configured to never delete old messages and just keep messages around indefinitely. See Section 10.2 for further discussion.

Validating accounts

The immediate validation of accounts is turned on by default on the msgstore channel so that it can, when presented with a popstore address, immediately check to see if it is valid or not (e.g., is it a valid recipient address, is the recipient allowed to receive new messages, etc.). This allows the various incoming mail streams to reject up front invalid messages for the popstore thereby obviating cases where the message is received only to then have to be bounced. Section 10.1.1 contains additional information on this account validation.

API

An API for sites who want to generate their own management, accounting, billing, logging, etc. facilities. In addition, agents which access the popstore or manipulate user accounts can be written using the API. See Chapter 12 for further details.

1.2 Licensing

Although installed as part of the base PMDF-MTA product, the PMDF popstore is licensed separately. The popstore can, however, be used without a license: sites without a PMDF-POPSTORE license can create up to ten popstore user accounts plus a default account. A PMDF-POPSTORE license enables a site to create more than ten user accounts.

From the interactive command line management utilities, the show -count_users command (SHOW/COUNT_USERS on OpenVMS) can be used to display the number of currently defined accounts as well as the limit allowed by your license. The web-based management utility can also display this information; see the "License limits" link in the menu at the top of the main page.

1.3 Accounts

Of particular importance is the account name and password. The account name specifies the mailbox for the popstore account. That is, if the popstore has the domain name sample.example.com then a popstore account with the name jdoe would have the e-mail address

jdoe@sample.example.com

Also, the popstore supports the concept of subaddresses. If an address contains a plus sign, +, in the local part then the plus sign and any characters to the right of it up to the at sign are ignored. For instance, the user jdoe can want to identify mail they receive from the HBD mailing list by subscribing themselves to that list with the address

jdoe+hbd@sample.example.com

Mail coming in to the popstore with that address will then be delivered to the account jdoe and not jdoe+hbd.

POP users access the mail stored for their account by supplying to their POP3 client their popstore account name and password. The rules for account names and passwords are given in Section 1.3.1 and Section 1.3.2.

Note that there is a special account --- the default account --- which is created at the time that the popstore is configured. The settings for the default account serve as account defaults. When new accounts are created, their defaults are copied from the default account.

The complete list of account fields is given in Table 1-1. Their interpretation and usage are made clear throughout the remainder of this manual.

Table 1-1 popstore Account Fields

Field name	Description
version	Data structure version number indicating what revision of the popstore account data structure is used by the account.
store_type	Storage type: popstore or MessageStore.
flags	Account usage flags.
ulen	Length in bytes of the value stored in the username field.
plen	Length in bytes of the value stored in the password field. The value stored in the plen field is stored in an encrypted form.
olen	Length in bytes of the value stored in the owner field.
slen	Length in bytes of the value stored in the private field.
username	The account's name. Maximum length of this field is 32 bytes.
password	The account's password. Maximum length of this field is 32 bytes. The value stored in this field is encrypted.
owner	Information about the account's owner. Maximum length of this field is 40 bytes.
private	Site-defined data field. Maximum length of this field is 64 bytes. The contents of this field can be set through the API or either of the management interfaces.
quota	The account's primary storage quota measured in bytes. A value of zero indicates unlimited storage quota.
return_after	How long to retain messages in the popstore before returning them.
overdraft	The account's overdraft quota measured in bytes.
last_billing	Time when billing information for the account was last generated. When the account is created, this value is set to the creation time of the account.
total_connections	Total number of connections made to the account.
last_connect	Time when the user last connected to the account with a POP3 client.
last_pwd_change	Time when the user last changed their password.
last_disconnect	Time when the user last disconnected from the account with a POP3 client.
total_connect	Total time, in seconds, spent connected to the account.
past_block_days	Storage block days for previously stored and since deleted or returned messages. Does not include storage values for messages currently being held in the store.
past_block_days_remainder	Roundoff from the past_block_days field. The roundoff is measured in units of byte minutes.
message_count	Count of messages presently stored for the account.
quota_used	Total size in bytes of the messages presently being stored for the account.
received_messages	Cumulative number of messages which have been stored for the account.
received_bytes	Cumulative number of message bytes which have been stored for the account.
glen	Length in bytes of the value stored in the group field.
group	Management group to which this account belongs. Maximum length of this field is 16 bytes.

1.3.1 Account Naming

1.3.1.1 The Preferred Naming Scheme

With this setting,

• usernames are case-sensitive,
• can contain any printable UTF-8 characters except for

  / @ % +

  and,

• must not begin with an underscore, _.²

- _ .

If you want to have case-insensitive behavior, create all your accounts using lower-case names, and use

$\$U$_

in appropriate domain rewriting rules to convert usernames to lower case on message delivery. Also, set TRANSLATE=ASCII-NOCASE in your security.cnf file to convert login usernames to lower case.

1.3.1.2 The Default Naming Scheme

In the default scheme, account names can be up to 32 characters long and can contain any of the characters from the set

0 1 2 3 4 5 6 7 8 9 . _ - a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Since account names are treated as being case insensitive, the names Anne and anne specify the same account.³ Account names should not begin with an underscore character, "_".²

1.3.1.3 Another Obsolete Naming Scheme

In this naming scheme, account names can contain any printable character in the DEC MCS or any one of the ISO Latin 1 through Latin 9 character sets (ISO 8559-1 through ISO 8859-9). That is, any character with decimal ordinal value in the ranges 33---126 or 161---255 (0x21---0x7E or 0xA1---0xFF in hexadecimal). On UNIX and NT platforms, the account names can be up to 32 characters long; on OpenVMS platforms up to 19 characters. Note that the names can contain spaces and punctuation characters. Again, the names are treated as being case insensitive. The account names should not begin with an underscore character, _, or contain plus signs, +.².

When a name is specified, it is converted to an account name using the following five steps:

1. All control characters are removed from the name. That is, all characters with decimal ordinal values in the ranges 0---31 or 127---159 are removed (0x00---0x1F or 0x7F---0x9F in hexadecimal).
2. All leading and trailing white space characters are removed. White space characters are the characters SPACE (0x20), TAB, (0x09), and non-breaking space (0xA0).
3. Each internal white space character is replaced with a SPACE character.
4. Consecutive SPACE characters are replaced with a single SPACE character.
5. All uppercase characters are converted to lowercase.

The choice of character set is specified with the USERNAME_CHARSET option as described in Chapter 3.

1.3.2 Account Passwords

1.3.2.1 Password Location

1. the user's popstore profile provided that the PWD_ELSEWHERE usage flag is not set for that profile;
2. or the PMDF password database if an entry exists for the user;
3. or the operating system's password database if an entry exists for the user.

When a password is set or changed for a popstore user, it will by default be

1. added or changed in the user's popstore profile provided that the PWD_ELSEWHERE usage flag is not set for that profile;
2. and changed in the PMDF password database provided that a previous entry for the user already exists;
3. and changed in the operating system's password database provided that an entry for the user already exists.

The above actions are the default behavior---the behavior when PMDF's authentication facilities are using their default configuration. Those facilities can be reconfigured to act differently and even to use other repositories of password information (e.g., an LDAP server or other authentication server). For more information on PMDF's authentication facilities, see the "Connection Authentication and Password Management" chapter of the PMDF System Manager's Guide. When using those facilities, the popstore uses a service name of POP. When a password is stored with the account's profile, the plain text form of the password is encrypted and stored. Such passwords are

• case sensitive,
• can contain any bytes (0x00---0xff), and
• can be up to 32 bytes long.

To store a password in an external source, mark the account with the PWD_ELSEWHERE usage flag. This tells the popstore that the password is stored externally. See also the information on password migration in the PMDF System Manager's Guide.

The POP3 server uses PMDF's authentication services and as such supports plain text as well as APOP, CRAM-MD5, and DIGEST-MD5 password authentication. Moreover, the POP3 server supports SASL (RFC 2222).

POP users can change their account passwords with PMDF's poppassd server, as described in Section 11.3, or with the web-based user interfaces described in Chapter 5. The poppassd server implements an ad hoc password changing protocol employed by several popular POP3 clients such as Eudora.

1.3.2.2 Using the Operating System's Password Database

1. the PMDF password database if an entry exists for the user;
2. or the operating system's password database if an entry exists for the user.

1.3.2.3 Password Security

• The time that a password is changed is recorded. This information is displayed in the output of the command line management utility and the web-based management interface.
• The popstore has support for password expiration, minimum password length, and basic "reasonableness" checks. These features are enabled by using the options PASSWORD_LIFETIME, PASSWORD_MINIMUM_LENGTH, and PASSWORD_REASONABLENESS, respectively.
• Sites may also apply additional validation or "reasonableness" checks through the VALIDATE_PASSWORD subroutine (see Section 14.3).
• Accounts can be marked "pre-expired", which forces the user of the account to change the password immediately. Note that "pre-expired" is only meaningful if password expiration has been enabled. An account can be marked pre-expired, or marked not pre-expired, using either the command line or web-based management interfaces.
• A utility is available to go through all accounts and mark them as not pre-expired. To run the utility, for example on unix:

  # /pmdf/bin/unpreexpire

  Note As soon as password expiration is enabled, all accounts created by versions of PMDF prior to V6.2-1 are automatically pre-expired.

1.3.3 Account Quotas

Each account has two storage quotas: a primary storage quota and an overdraft quota. This two-quota scheme is used for efficiency purposes. With most message transfer protocols, the size of an incoming message is not known upfront and a message has to be received in its entirety in order to determine its size. Use of an overdraft quota allows PMDF to temporarily refuse to accept incoming messages for an over quota user. How so? If a user has just one quota limit, then odds are they will never quite attain it. For instance, if they have 100 spare bytes of storage they are not over quota and so the SMTP server will accept new messages for the user. However, most received messages will be larger than 100 bytes and will thus need to be bounced. This will continue until the user receives enough small messages to exactly use up those remaining 100 bytes. Once they have exactly consumed their quota, the SMTP server can now immediately refuse further messages without the need to first receive the message and see how large it is. But this is only possible once the user has exactly consumed their quota. An alternative is to have a "fudge" factor of some form. Once the user is somehow close to their quota, the server refuses additional messages. The popstore's overdraft quota is such a fudge factor. A user's quota is really a quota range with a lower limit given by their message storage quota and the upper limit given by the sum of their message storage quota and their overdraft quota.

All that having been said, note that by default PMDF will not reject incoming messages for an over quota user. By default, PMDF accepts the messages and holds onto them until either the user has available quota to receive the messages or the messages "times out" and are returned by PMDF's message bouncer. To have PMDF reject incoming messages for an over quota user, specify REJECT_OVER_QUOTA=1 in the popstore option file as described in Section 3.4. When REJECT_OVER_QUOTA=1 is specified and a user is over quota, the message copy for the over quota user will be rejected with a temporary error message.

If desired, accounts can be granted unlimited storage quota as denoted by a primary quota value of zero.

1.3.4 Management Groups

Group names are case-insensitive and can be zero to sixteen bytes long. Group names are shared across all user domains. When you create a new account and do not specify a group name for the account, the account is placed in the same group as the default account. By default, the default account is not in any group --- it has a group name of zero length. This group with a zero length name is referred to as the world group.

There are two primary uses for management groups:

1. Accounting: For instance, listing or billing all accounts within the same management group.
2. Management: Allowing a privileged account to only manage a subset of the popstore accounts.

The actual details behind defining groups and assigning group names to accounts is documented in Chapters 4 --- 7 as part of the discussions of the various account management commands. Note that only two classes of users can create, delete, or modify group definitions. These classes are (1) users with operating system privileges, and (2) popstore users with privileged account which themselves are in no group [and thus are in the world group]. The first class can use the interactive command line management utilities; the second class can use either the interactive command line or web-based management utilities.

Groups can be nested. That is, a group can contain subgroups and those subgroups can contain further subgroups. An account is contained in the group G if either (1) the account's group name is G, or (2) the account's group is a subgroup (nested arbitrarily deep) of the group G. The world group --- the group with zero length group name --- is a distinguished group: it implicitly contains all other groups.

You can visualize typical group hierarchies as an inverted tree. A hypothetical example with nine groups is shown in Figure 1-1. The root of the tree is the world group, which contains all other groups. The staff and faculty groups have no subgroups. The students group, however, has five subgroups.

Group hierarchical structure need not be a tree: in the language of graph theory, loops are allowed. For instance, a group can be a subgroup of more than one group. Modifying the example of Figure 1-1, it is possible for grad to also be a subgroup of staff.

Figure 1-1 Example management groups

The ability to nest groups is particularly useful in regards to account management. An account with management privileges can manage any account within the same group as the privileged account itself. Thus, if a privileged account has a zero length group name, then that account can manage any and all accounts. If, however, the privileged account is in a group with a non-zero length group name, then that privileged account can only manage accounts contained within the same group. For instance, a privileged account in the students group can manage any account in that group; i.e., any account with group name students, class97, class98, class99, class00, or grad. A privileged account in the group class97 can only manage other accounts in the group class97. If you want to have an account which can manage both staff and faculty but not students, then just create a new group named, for instance, dean and make staff and faculty be subgroups of that new group. Then, make the group name for the privileged account be dean.

If you want to allow an account to manage all accounts yet be in a named group such as manager, then create a group named manager and make the world group be a subgroup of the manager group.

1.3.5 User Domains

The popstore user user in the user domain host has the e-mail address user@host.⁶ The delivery channel, which can deliver mail for several different user domains, determines which domain or domains a message should be delivered to by examining each envelope recipient address. The non-legacy POP server determines which user domain a client is in from either the username presented by the client or from the PORT_ACCESS mapping table. In regards to the former, the client must present a username of the form user%host or user@host.⁷ In regards to the latter, consult the PORT_ACCESS mapping table documentation in the PMDF System Manager's Guide. Use of that table allows selection of the user domain to be based upon such information as the client's source IP address or the TCP port which the client has been configured to use for POP service.

When managing the popstore via the Web-based management utility, the user domain to manage is specified by including it in the URL; for example, to manage the example.org user domain, use the URL

http://host:7633/popstore/example.org/admin.html

When using user domains, there will be a special user domain referred to as the default domain. The official host name for the delivery channel will be mapped to the default domain. Moreover, a privileged management account in the default domain which is in no group can manage any account within the popstore regardless of user domain. To make this a little more clear, consider the channel definition

popstore defragment holdexquota example.com example.org

In the above, the host example.com is the channel's official host name and therefore identified with the popstore's default user domain; the host example.org is identified with the example.org user domain.⁸ Mail to jdoe@example.com is delivered to the account jdoe in the example.com user domain. Mail to jdoe@example.org is delivered to the separate account jdoe in the user domain example.org. Accounts associated with the example.com host are managed by managing the default user domain; accounts for the sample.com host are managed via the example.org user domain.

See Sections 6.13 and 7.13 for detailed information on creating a user domain and managing accounts within it.

1.3.6 Account Usage Flags

DISMAIL

The DISMAIL flag is used to prevent an account from receiving new mail messages. When this flag is set for an account, new messages are rejected and returned to their sender. The account owner can, however, read any existing messages they might have unless the account is also flagged with the DISUSER flag.

DISUSER

The DISUSER flag is used to deny access to an account. The account can, however, continue to receive new messages unless it is either over quota or also flagged with the DISMAIL flag. When the user attempts to access their account, they will be met with an "account disabled" error message.

LOCKPWD

The LOCKPWD flag prevents users from changing their account's password. The password can only be changed by a user with the management privilege or operating system privileges.

MANAGE

Accounts with this flag can use the web-based interface to manage the popstore. In addition, users with unprivileged login accounts to the platform running the popstore can manage the popstore using the command line interface when their popstore account has the MANAGE flag set. This is accomplished through the command line interface's LOGIN command. Section 1.3.7 for further details.

MIGRATED

This is a flag used by the PMDF's migration utilities to track whether or not migration from an external source to the popstore has been completed for a given popstore user account.

PWD_ELSEWHERE

This flag tells the popstore that the user's password information is stored externally, outside of the popstore. The popstore's authentication mechanisms use this flag when determining how to authenticate a user password. See Section 1.3.2 for further details.

With the exception of the MANAGE and NOMANAGE flags, these flags can be set or cleared with either the web-based or command line management interfaces. As a security precaution, the MANAGE and NOMANAGE flags can only be manipulated through the command-line interface.

Note that PMDF itself has a variety of other access control mechanisms such as the SEND_ACCESS mapping table to control access at the envelope address level and the PORT_ACCESS mapping tables to control access at an IP level. See the PMDF System Manager's Guide for details on these and other access mapping mechanisms provided by PMDF.

1.3.7 Privileged Accounts

Privileged accounts can only manage other accounts within the same management group and user domain. If a privileged account is in the world group (i.e., is not in any group or is in a group which explicitly contains as a subgroup the world group), then it can manage any popstore account within the same user domain. However, a privileged popstore account which is in the world group and is in the default user domain can manage all accounts within all user domains and groups.

1.3.8 Bulk Loading

1.3.9 Account Storage

Read and write access to these files is controlled using private locks. As such, they should only be accessed using the popstore API routines documented in Chapter 12.

Each profile file has a base size of 256 bytes plus 40 bytes per message stored for the user. Presently, the profile files are interchangeable amongst the different platforms on which the popstore runs. That is, for instance, it is presently possible to move the popstore from one platform to another without the need to reformat the profile files.⁹ See Section 8.1 for complete details on migrating the popstore to another platform.

1.4 Messages

When PMDF receives a message for the popstore, the message is queued to the popstore's inbound delivery channel. That channel then processes each queued message and stores the resulting messages in the popstore's message store. A single message copy is stored for all recipients of a message. Once a message has been deleted by each recipient, it is deleted from the store itself.

Message storage quotas are set on a per-user basis via the quota and overdraft quota account settings. See Section 1.3.3 for further details.

Users access their stored mail messages via their POP3 client and the popstore's POP3 server. Ideally, users download their messages and delete them from the store itself. By default, if a message is not deleted within a fixed number of days, the message will be deleted silently. Should one or more of the recipients not have read the message, a non-delivery notification is sent to the message's originator prior to deleting it.

Popstore administrators can delete message files using either of the two management utilities. Optionally, when a message file is deleted, a non-delivery notification can be sent to the originator of the message. The non-delivery notification will state which recipients had not read the message.

The actual message files are binary files stored in a ready-to-download format. On UNIX systems, these files are kept in the directory tree specified by the PMDF_POPSTORE_MESSAGES option in the PMDF tailor file; on NT systems, the PMDF_POPSTORE_MESSAGES registry entry is used; and, on OpenVMS systems, in the PMDF_POPSTORE_MESSAGES: directory tree. Read and write access to these files is controlled using private locks. The files should only be accessed using the popstore API routines as documented in Chapter 12.

The size of each message file varies depending upon the amount of envelope information and message content which must be stored. Presently, the message files are interchangeable amongst the different platforms on which the popstore runs. That is, for instance, it is presently possible to move the popstore from one platform to another without the need to reformat the message files. While Process Software hopes to maintain this level of portability in the future, it can not be possible, at which point a conversion utility will be provided.

1.5 Forwarding Mail

Forwardings are recursive. That is, if a popstore address has a forwarding which in turn points to another popstore address, then that new address will also be checked for a forwarding. In addition, a forwarding can point to more than one address. That is, a forwarding can forward to multiple addresses and, moreover, some of those addresses can themselves be forwarded.

Forwardings are established, examined, and removed through either of the management interfaces. The forwardings are stored in an ordinary PMDF CRDB database --- the PMDF_POPSTORE_FORWARD_DATABASE. If desired, the database can be manipulated using ordinary PMDF tools as well as through the PMDF API.

Messages destined to the popstore can explicitly defeat any forwarding by prefixing the address with an underscore character, _. For instance, if the account "jdoe" has mail forwarded elsewhere, then a message sent to the address _jdoe@example.com will bypass that forwarding and be delivered to the "jdoe" account. To explicitly forbid such bypassing, the DISMAIL usage flag can be set for the account in which case mail to _jdoe@example.com will be returned as undeliverable.

Finally, note that it is the popstore delivery channel which actually effects mail forwardings. When processing message files, it checks each recipient to see if a forwarding exists. If it does, it then uses the forwarding address instead. If the forwarding address points back to the popstore, it then checks that address for a forwarding. This process is iterated up to ten times; an address which is forwarded internally more than ten times is deemed to be a forwarding loop and is rejected. If the forwarding address is external to the popstore, then a new message is enqueued with the forwarding address used as its envelope To: address.