Migrating to CommuniGate Pro

Users that access their mail accounts using any standard Internet Protocol (POP, IMAP) do not have to switch their mailer applications or change mailer settings - CommuniGate Pro supports not only the published mail access protocols standards, but most of unofficial protocol extensions, too.

Some users registered with the server OS may access their mail accounts using legacy mailer applications that read mailbox files directly. Since these mailers bypass Internet protocols, the CommuniGate Pro server has no control over those mailers.

The CommuniGate Pro Server keeps all user Accounts and Mailboxes inside its "base directory". On a properly configured system direct user access to the base directory is prohibited to ensure that Account Mailboxes and other Server files stay intact.

When you create a CommuniGate Pro Account for a local user who needs mail access via legacy mailers, select the Legacy INBOX option. In this case, the INBOX Mailbox will not be created inside the CommuniGate Pro base directory. Instead, the INBOX Mailbox location will be taken from the Domain settings.
Usually, you specify /var/mail/* or /var/spool/mail/* - the "standard" location where legacy mailers expect to see user Mailboxes.

When the Server accesses these Legacy Mailboxes, it uses the OS file-locking mechanism to synchronize its activity with legacy mailers.

Note: legacy mailers were not designed to support simultaneous access to mailboxes. They can destroy data in your mailbox if you open two legacy mailer sessions with the same mailbox and delete messages in one of the sessions. CommuniGate Pro cannot fix this, since these mailers bypass the server, it can only guarantee (using file locks) that legacy mailers do not destroy a mailbox while CommuniGate Pro is working with it.

For more details, see the Mailboxes section.

The default format for CommuniGate Pro Mailboxes is the BSD-type text mailbox format: a Mailbox is a text file with messages separated with an empty line and a line starting with the symbols From .

Since most legacy mail systems use this format, the existing mailbox files can be used when migrating to CommuniGate Pro. You should either copy the old mailbox files into the proper places in the CommuniGate Pro Account directories, or you can specify that some Accounts have Legacy INBOXes (see above), and the old mailbox files will be used "in place".

Note: when CommuniGate Pro stores a message in a BSD-type Mailbox, it adds additional fields to the separator (From) line. These fields are ignored by legacy mailers and mail servers, but they allow the CommuniGate Pro Server to keep the message status and unique message ID information. When you make your CommuniGate Pro Server use a BSD-type mailbox file composed with an old mail system, it issues warnings (Log records) about missing fields in the message separators. The Server still opens such mailboxes: it creates empty status flag sets for such messages, and it generates unique IDs on-fly. As users read, move, and/or delete old mail from their mailboxes, messages stored with the old mail system disappear, and the Server stops complaining when opening these mailbox files.

If your old mail server authenticated clients using the Unix OS account and passwords (the passwd and shadow files), you can simply enable the Use OS Password option for those accounts and the CommuniGate Pro Server will use the OS authentication procedures for them.

Since the OS Passwords are one-way encrypted, they cannot be used for secure SASL authentication methods. The following procedure can be used to allow migrated users to employ the secure SASL methods:

• enable the Use OS Password option either in the Default Account Settings or in the Account Template.
• specify an empty string for the CommuniGate Password in the Account Template.
• import all accounts without the Password field.

Users can connect to the newly created accounts using their old OS Passwords - i.e. the same passwords they used with the legacy mail system. When users try to modify their account passwords, the new passwords will be stored as CommuniGate Passwords. All users that have updated their passwords will be able to use the secure SASL authentication methods.

Sometimes you cannot use this method. For example, you migrate users from a different server, and you do not register them all with the Unix OS on the new server, but you do have the passwd file from the old server. In this case, you may want to enter the Unix-style (crypt-encrypted) passwords as the CommuniGate (internal) Passwords.

To make the CommuniGate Pro server process its internal password string as a U-crpt (crypt-encrypted) or other support encrypted password (see below), store the password in the Account Settings with one-byte binary 002 prefix. If you want to create a user test using the CLI interface, and the Unix (crypt-encrypted) password for that user is AslUzT1JkPsocc, then use the following CLI command:
createaccount "test" {Password="\002AslUzT1JkPsocc";}

If you create users by importing an account list from a text file, place the Unix passwords strings into the UnixPassword column, not into the Password column. In this case, the Loader will automatically add the binary 002 prefix to all password strings. If you create users using the LDAP Provisioning feature, specify the encrypted password as the unixPassword attribute.

Account Settings of the new accounts should specify one of the CommuniGate Pro password encryption methods (clear text or A-crpt). Users will be able to log in using their old Unix/encrypted passwords. When they update their passwords, new CommuniGate Password strings will be stored using the specified CommuniGate Pro password encryption method. All users that have updated their passwords will be able to use the secure SASL authentication methods.

Some servers produced by the Netscape and Software.com companies store user passwords using several encryption methods. When passwords are retrieved from those servers, they have the following form:
{method}eNcoDeD
or
$method$eNcoDeD
where method specifies one of the standard encryption methods, and the eNcoDeD string is an encrypted password (sometime - Base64-encoded).

CommuniGate Pro can use these passwords in the same way it uses the Unix-encrypted passwords, and they should be entered in the same way: you should use the binary 002 prefix in the CLI commands and/or you should place those passwords into the UnixPassword field of the Account Import file.

The following encryption methods are supported:

• {crypt} - the standard Unix crypt method.
• {WM-CRY} - the standard Unix crypt method (same as {crypt}).
• {MD5} - the MD5 digesting method (Base-64 encoded MD5 digest of the password string).
• {SHA} - the SHA1 digesting method (Base-64 encoded SHA1 digest of the password string).
• {NS-MTA-MD5} - the MD5-based method used in the Post.Office servers and old Netscape Messaging Servers (the eNcoDeD portion contains 64 hexadecimal digits).
• {SSHA} - the "salted SHA1" digesting method (Base-64 encoded SHA1 digest of the password and salt strings, followed with the salt bytes). This method is used in the Sun Directory Server application.
• {LANM} - the "LAN Manager" hash used in Microsoft Windows servers (the eNcoDeD portion contains 32 hexadecimal digits).
• {MSNT} - the "Microsoft NT" hash used in Microsoft Windows servers (the eNcoDeD portion contains 32 hexadecimal digits).
• $1$ - MD5-based password hash used in FreeBSD and some other Unix systems.
• $2a$ - BlowFish-based password hash used in OpenBSD and some other Unix systems.

The following is a sample Import file:

Name UnixPassword Password Type user1  YIdhkjeHDKbYsji Unix-crypt user2  {SHA}Ue4Erbim2TC7CmuukMOBejeytr2= SHA1-digested user3  {MD5}zverMUhsgJUIDjeytr2= MD5-digested user4  {crypt}YIdhkjeHDKbYsji Unix-crypt, same as for the user1 account user5  $1$VlPrB$vNjOAytB3W.j0bkkbaN2Z. BSD-type MD5-encrypted

You can use a CommuniGate Pro CLI script to automatically import all users and their passwords from the OS /etc/passwd file. See the CommuniGate Perl Interface site for a sample script.

If you are migrating from a sendmail-based mail system, you may find the following information useful:

The aliases file

The sendmail aliases file allows the administrator to redirect local mail to one or several addresses. Sendmail uses the term "alias" for too many different things, so you should select the proper CommuniGate Pro feature to implement different types of sendmail "aliases":

• Each account can have one or several aliases. Mail sent to any Account Alias name is routed to the Account itself. If the domain.dom Account john.smith has j.smith and smith aliases, mail sent to j.smith@domain.dom and smith@domain.dom addresses is delivered to the john.smith@domain.dom Account. When an Account is renamed, its Aliases automatically start to point to the new name, and when an Account is removed, all its Aliases are removed, too.
• Domain Forwarder objects allow you to redirect mail sent to a Domain address to any other address. The domain.dom Forwarder susan.smith can reroute all mail sent to susan.smith@domain.dom address to the susan@otherisp.dom address.
• Domain Group objects allow you to redirect mail sent to some Domain address to any set of addresses.
• The Router module allows you to redirect mail sent to a certain address to any other address. The Router Alias Record <*.smith@domain.dom> = Smith@domain.dom will reroute mail sent to john.smith@domain.dom and to susan.smith@domain.dom to the Smith@domain.dom address.
• The Account Rules allow the administrator and/or the users themselves to redirect/forward/mirror all or certain mail to one or several addresses.
• The Server-Wide Rules allow the administrator to redirect/forward/mirror all or certain mail to one or several addresses.
• The shared or Foreign Mailboxes feature allows a user to grant access to a Mailbox to other users; in many cases a shared Mailbox is a much better alternative to mail distribution.
• The LIST module provides a very powerful mailing list distribution mechanism.

procmail processing

The Server-Wide, Domain-Wide, and Account-Level Automated Rules allow administrators and users to perform automatic mail processing and filtering using the powerful condition checking and processing operations built into the CommuniGate Pro Server.
For the situations where messages should be processed using an external filter or processor, the Execute Automated Rules operation can be used to start the specified external program as a separate OS task (for example, the Rules can be used to process all or certain incoming messages with the same procmail program).

The Post.Office product stores account names, passwords, and other information in its account database. The special Post.Office Migration Utility can be used to retrieve that information and to store it in a tab-delimited file that can be used with the CommuniGate Pro WebAdmin Account Import function.

The List Migration script allows you to copy mailing lists and their subscriber sets from Post.Office to CommuniGate Pro.

The Netscape (iPlanet) Messaging server stores account names, passwords, and other information in a Directory server "subtree". Use regular LDAP tools to export the Directory subtree into an LDIF file. The special Netscape Migration Script can be used to convert the account information from the LDIF format into a tab-delimited file that can be used with the CommuniGate Pro WebAdmin Account Import function.

The IMail product stores account names, passwords, and other information in its account database. The special IMail Migration Utility can be used to retrieve that information and to store it in a tab-delimited file that can be used with the CommuniGate Pro WebAdmin Account Import function.

If you want to move your users from a CommuniGate for MacOS server, you can build the account list file using the CommuniGate/MacOS extractor utility.

If you want to move your users from a Stalker Internet Mail Server (SIMS), you can build the account list file using the SIMS extractor utility.

The special Exchange Migration Utility allows you to retrieve user lists from an Exchange server and to create those users in CommuniGate Pro Domains. The utility copies all user folder data (mail, calendaring, contacts, etc.) converting data and address formats "on-the-fly".

The utility also extracts the Exchange Global Address Book and converts it into an LDIF file that can be imported into the CommuniGate Pro Directory.

The utility requires an MS Windows workstation to run.

When migrating from other mail servers, you may want to copy all messages from an account on the old server to the already created account on the new server.

The CommuniGate Pro software package includes the MovePOPMail program. This program connects to the old POP server, logs in, retrieves all messages, and submits it to the new SMTP server:

MovePOPMail [--verbose] [--delete] [--notimeout] POPserver POPname POPpassword SMTPserver SMTPrecipient

POPserver

The IP address of the old (source) POP3 server; if that POP server operates on a non-standard TCP port, you can specify the port number using the colon symbol: 192.0.2.3:111 - POP server at the 192.0.2.3 address, port 111.

POPname

The POP account name - i.e. the name of the source account on the POP server.

POPpassword

The POP account password.

SMTPserver

The IP address of the new (target) SMTP server; if that SMTP server operates on a non-standard TCP port, you can specify the port number using the colon symbol: 192.0.2.4:26 - SMTP server at the 192.0.2.4 address, port 26.

SMTPrecipient

The address to send the retrieved messages to. Usually - the account name on the new server.

--verbose

An optional parameter. When specified, the progress information is sent to the standard output.

--delete

An optional parameter. When specified, the program deletes all retrieved messages from the POP account.

--notimeout

An optional parameter. When specified, the program increases the SMTP and POP operation time-out values from 20 seconds to 1 hour.

Sample:

When migrating from other mail servers, you may want to copy all mailboxes and all messages from an account on the old server to the already created account on the new server.

The CommuniGate Pro software package includes the MoveIMAPMail program. This program connects to the old and new IMAP servers, logs into both, retrieves the list of mailboxes in the old account, creates all missing mailboxes in the new account, and copies all messages from mailboxes in the old account to the mailboxes in the new account. The program also copies the list of "subscribed mailboxes", and the mailbox ACLs (if supported).

MoveIMAPMail [flags] OldServer oldName oldPassword NewServer newName newPassword

oldServer

The IP address of the old (source) IMAP4 server; if that IMAP server operates on a non-standard TCP port, you can specify the port number using the colon symbol: 192.0.2.3:144 - IMAP server at the 192.0.2.3 address, port 144.

oldName, oldPassword

Strings to use when logging into the source IMAP server.

newServer

The IP address of the new (target) IMAP4 server; if that IMAP server operates on a non-standard TCP port, you can specify the port number using the colon symbol: 192.0.2.5:145 - IMAP server at the 192.0.2.5 address, port 145.

newName, newPassword

Strings to use when logging into the target IMAP server.

Flags is zero, one or several optional parameters:

--verbose

An optional parameter. When specified, the progress information is sent to the standard output.

--list search

An optional parameter. When specified, the following search string is used to find all mailboxes in the source account. Some IMAP servers show the entire user directory or even system directory if the default search string "*" is used. Consult with your old IMAP server manual to learn the search string to use.

--source prefix

An optional parameter. When specified, the following prefix string is used as the first parameter of the "LIST" command (see above). If the mailbox names produced with the LIST command start with the specified prefix, the prefix is removed from the name when the mailbox is created on the target server.
This feature allows you to copy a subtree of the source account mailbox tree into the "top" level of the target account mailbox tree. If the source account contains mailboxes abc/mail1 and abc/mail2, then --source abc/ parameter can be used to copy just these 2 mailboxes and to create them as "mail1" and "mail2" mailboxes in the target account.
If the source server is CommuniGate Pro, this parameter can be used to copy all mailboxes from any account, using the postmaster name and password: --source '~username'/
See the --target option below.

--target prefix

An optional parameter. When specified, the following prefix string is appended to all mailbox names sent to the target server. For example, if the target server is CommuniGate Pro, you can specify the postmaster credentials in the parameters (instead of the username credentials), and use the
--target '~username/'
parameter to copy mailboxes to the username account. This can be useful when you do not know the username account password.

--skipMailbox mailboxName

An optional parameter. When specified, the mailboxName mailbox is not copied.
This parameter can be specified more than once, to exclude several mailboxes.

--notimeout

An optional parameter. When specified, the program increases the IMAP operation time-out value from 20 seconds to 1 hour. You may want to specify this option when your copy mail from slow servers.

--delete

An optional parameter. When specified, the program deletes the retrieved messages from the source account.

--nosubscription

An optional parameter. When specified, the program does not copy the mailbox subscription list to the target account.

--subscribed

An optional parameter. When specified, the program copies only those mailboxes that are listed in the source account mailbox subscription list.

--fetchRFC822

An optional parameter. When specified, the program uses the RFC822.PEEK attribute instead of the BODY.PEEK[] attribute when it sends IMAP FETCH commands to the source server. Use this attribute when the source server is too old and does not support the BODY.PEEK[] FETCH attribute.

--byOne

An optional parameter. When specified, the program fetches messages from the source IMAP mailbox one by one; otherwise it fetches all messages at once. Use this parameter when the source server fails to retrieve all mailbox messages with a single FETCH command.

--noACL

An optional parameter.When specified, the program does not copy the mailbox ACL (access control lists) to the target account.

--copyMailboxClass

An optional parameter. Can be specified if both source and target servers are CommuniGate Pro servers. When specified, the program copies the mailbox classes ("calendar", "contacts", etc.)

--fixLongLines number

An optional parameter. Can be specified if the source server has messages with extremely long text lines. These lines will be separated into several lines so no message line in a target server mailbox is longer than number bytes.

--proxyAuth username

An optional parameter. When specified, the 'PROXYAUTH username' command is used with the source IMAP server. Use this command to login to the source server as admin to retrieve mail from an account whose password you do not know, if your source server supports the PROXYAUTH IMAP command.

--renameMailbox oldMailboxName newMailboxName

An optional parameter. When specified, the oldMailboxName mailbox name in the source account is automatically translated into newMailboxName name in the target account.
This parameter can be specified more than once, to rename several mailboxes.

--renameMailboxList file.txt

An optional parameter. When specified, a list of mailboxes renaming pairs is read from the specified text file. Each file line should contain pairs of an old and new mailbox names separated with the tabulation symbol. When a source account mailbox has a name listed as an "old" name in the file, the mailbox messages are copied into the target account mailbox with the name specified as "new" on the same file line.

--filter [before | after ] "dd-mmm-yyyy[ hh:mm:ss]"

An optional parameter. When specified, the program copies only messages with the INTERNALDATE before or after the specified date. The hh:mm:ss part is optional, if it is not specified 00:00:00 is assumed.

Sample:

Note: if a mailbox name in the source account ends with symbols .mbox or .mdir, the mailbox name in the target account will end with the -mbox or -mdir symbols.

Note: if a mailbox name in the source account starts with the symbol . or ~, the mailbox name in the target account will start with the _ symbol.
Leading and trailing spaces, the \ and # symbols in the source account mailbox names are replaced with the _ symbols when the target account mailbox names are composed.

After you have created the accounts on your new CommuniGate Pro Server, you may want to copy mail from all mailboxes on the old server to the accounts on the new server.

The CommuniGate Pro software package includes the MoveAccounts program. This program uses a tab-delimited text file that contains account names and passwords. It can be the same file you have used to Import Accounts to a CommuniGate Pro domain.

The program scans the file and uses either the MovePOPMail or the MoveIMAPMail program to move messages for each account. These programs should be located in your current directory.

MoveAccounts [--POP | --IMAP] file sourceServer targetServer [suppl_parameters]

--POP, --IMAP

Parameters that specify whether MovePOPMail or MoveIMAPMail program should be used. If none is specified, the MovePOPMail program is used.

file

The name of a tab-delimited file that contains account names and passwords.

sourceServer

The IP address of the old (source) server (POP or IMAP); can include the port number.

targetServer

The IP address of the new (target) server (SMTP or IMAP); can include the port number.

suppl_parameters

Optional parameters (such as --verbose, --delete, --notimeout, --list search, etc.) passed to the MovePOPMail or MoveIMAPMail program.

The first line of the file should contain the data field names. The fields with names Name and Password must be present.

If the field NewName exists, it is used as the SMTPrecipient parameter when the MovePOPMail program is started, or as the newName parameter for the MoveIMAPMail program. Otherwise the same Name field data is used.

If the --IMAP parameter is specified, the program checks if the NewPassword field exists. If it does, the data in that field is passed as the newPassword parameter to the MoveIMAPMail program. Otherwise, the same Password field data is used.

All fields with other names are ignored.

The following is a sample AccountList file:

Name Limit Password john 10K j27ss#45 jim 120K dud-ee george 31M mia#hj!

If you cannot obtain the clear-text passwords for all accounts you want to copy, and the old server is using the Unix /etc/passwd or /etc/shadow file, follow these steps:

• Find the OS file that contains the encrypted user passwords: /etc/passwd, /etc/shadow, or /etc/master.passwd (see your OS documentation). We will refer to that file as /etc/shadow.
• Create a backup copy of the /etc/shadow file.
• Find the /etc/shadow record that contains information for your own account or any other account you know the clear-text password for. Let us say that this known unencrypted OS account password is mypassword, and the encrypted password you see in the /etc/shadow account record is FU3jjF/gkJJdA.
• Edit the /etc/shadow file so all account records will contain FU3jjF/gkJJdA in their encrypted password fields.
• Open the CommuniGate Pro Default Account Settings page for the domain you are migrating. Enable the Use OS Passwords option and check that the OS UserName option is set to *.
• Create the AccountList file that contains the account names in the Name field, and the string mypassword in the Password field.
• Copy all account mailboxes from the old server to the new server using the MoveAccounts command. The command will successfully log into both servers, since all accounts on both servers accept the string mypassword as the account password.
• Restore the /etc/shadow from the backup copy.
• Disable the Use OS Password setting in the CommuniGate Pro Default Account Settings, if you do not want to use OS Passwords for your CommuniGate Pro Accounts.

Use this alternative migration method when the password switching method explained above cannot be used, and/or the user names and passwords cannot be retrieved from the old server.

The method is based on the External Authentication feature of the CommuniGate Pro server.

Download, install, and tailor the migration script, and configure CommuniGate Pro to use this script as the CommuniGate Pro External Authentication program.

Create the target CommuniGate Pro Domain, and enable the Consult External for Unknown Domain settings. Disable the Use CommuniGate Password option and enable the Use External Password option in the Account Template.

When a user attempts to connect to a non-existent Account, or when CommuniGate Pro receives a message for non-existent Account, the External Authentication script is called. The script connects to the old server using the SMTP protocol and checks if the account with the same name (same address) exists on the old server. If the old server does not reject the address, the script creates the Account with this name in the CommuniGate Pro Domain. Then the CommuniGate Pro Server delivers the message to this newly created Account.

When a user connects to the CommuniGate Pro Server, the user mailer sends the user name and the user password in the plain text form. Because the CommuniGate Pro Account has the Use CommuniGate Password option disabled, and the Use External Password option enabled, the External Authentication script is called. The script connects to the old server using the POP or IMAP protocol and checks if it can log into the old server with the provided user credentials.

If the old server accepts the specified password:

• the script uses the CommuniGate Pro CLI:
  • to set the specified password as the CommuniGate Password for this Account,
  • to switch off the Use External Password option for this Account,
  • to switch on the Use CommuniGate Password option for this Account.
• the script starts the MoveIMAPMail or MovePOPMail programs to copy the account mailboxes from the old server to the CommuniGate Pro server, or saves the name/password pairs into a text file which you can later use with the MoveAccounts program (this is a configurable option).

After the first successful login, the correct password will be set as the new CommuniGate Password, and all Account mail will be copied from the old server.

After all old server users have successfully connected to the CommuniGate Pro server at least once, all their Accounts will be created and have the correct CommuniGate Passwords set. Then you can disable the External Authentication script and retire the old server.

Very often it is essential to switch to the new server without any interruption in the services you provide.

If you install the new CommuniGate Pro server on the same system as the old mail server, you should:

• switch CommuniGate Pro SMTP receiving to port 26, so it will not interfere with your old server SMTP activity.
• switch CommuniGate Pro POP service to port 111, and IMAP service to port 144, so they will not interfere with your old server POP/IMAP activity.
• Configure CommuniGate Pro and create Domain Aliases and/or full featured Secondary Domains.
• Create some test accounts in the main and secondary domains and check that you can log into those accounts using WebUser Interface.
• Check that you can send mail from those accounts using the WebUser Interface: mail to other test accounts in the created domains and mail to other servers should be delivered correctly.
• If you have a POP or IMAP client that allows you to specify a non-standard port number, check that you can log into the test accounts on the POP port 111 and IMAP port 144.
• Create tab-delimited file(s) with names, passwords, and other attributes of your existing accounts and use the Account Import feature to create all accounts on your new server.

All this can be done while your old server is still active.

Now, you should stop your old server activity by either changing its port numbers to non-standard values, or by disconnecting it from the external network.

Use the AccountMove program to copy all messages from your old server to CommuniGate Pro.

When all messages are copied, change the CommuniGate Pro SMTP port number back to 25, POP port number - to 110, and IMAP port number to 143. Now CommuniGate Pro will operate as your mail server, without any interruption in the services you provide.

You may want to keep the old server running for several hours - in case its queue contained some delayed outgoing messages. Just check that the old server does not try to use the standard ports.

If you create several Secondary domains in CommuniGate Pro, you may want to move accounts from your old server(s) to a Secondary CommuniGate Pro Domain, not to its Main Domain.

In this case, you should add the NewName field to your account list file, and copy all names into that column, adding the @domainname string to each name.

If you use the IMAP protocol to move messages between the servers, you may use a simpler method:

• If the target domain has an IP address assigned to that domain, use that address in the mail copying programs: all non-qualified names provided on connections established with that address are interpreted as names in that domain. See the Access section for the details.
• If the target domain does not have an assigned IP address, temporarily assign the IP address of the main domain to that secondary domain. Move the messages, and remove the IP address from the list of addresses assigned to that domain.