Configuring Mutt to work with Duo for uwaterloo.ca Mail

So the University of Waterloo is steadily but surely moving from internally-hosted Microsoft Exchange to Office365. As an alumnus my number has come up. Soon my UW Mail will be in The Cloud (tm), but I want to continue using Mutt to read my mail. OWA (Outlook Web App, aka webmail) exists and it is okay for searching mail, but it is awful as a daily mail client.

Unfortunately this move means I have to authenticate to the Office 365 using Duo, the UW Multi-factor Authentication system. This looked like a big problem. For a while I tried using Davmail, which allowed me to authenticate to Duo, but was unusably slow. Fortunately, recent versions of Mutt have OAuth2 support, which is what you need to connect to Office365.

tl;dr

Follow the instructions here: https://www.vanormondt.net/~peter/blog/2021-03-16-mutt-office365-mfa.html . As of this writing they work. I will call these the reference instructions. These instructions are worth reading even if you also follow this guide.

Unfortunately for me, I had to work through a bunch of sub-steps in order to get through the above instructions. I will document that yak-shaving below. I will try to be explicit about what information goes where, and in what format, since this is where I tended to get confused.

Configuring Mutt to work with Duo for uwaterloo.ca Mail
Sidebar!

Terminology

There are a bunch of different names to worry about:

First is my University of Waterloo email address. Let's call that pntestin@uwaterloo.ca, because that was the test account the nice people at IST set up for me so I could do testing before my account was migrated. (Thanks, IST.)
Second is an email address associated with my GPG identity. This does not need to be the same as my UW email. Let's use pnijjar@example.com for this.
There is an OAuth2 token file you will need to create. Let's use ~/.mutt_token for this.

My netbook is running Debian Bullseye (11.x). It has no fancy desktop environment.

My laptop is running Ubuntu (actually Xubuntu) Focal (20.04).

There will be a lot of discussion of a client ID and client secret, which are part of an App Registration. I document the Thunderbird client ID and client secret below.

IST is Information Systems and Technology, the IT group that administers email at the university. UW is the University of Waterloo.

Overview

If a mail client supports OAuth2 then (supposedly) it works with Duo. Mutt supports this as of 2018 or so, but the reference instructions say we need Mutt version 2.0.0 or later to support a imap_oauth_refresh_command function in .muttrc .

We also need a Python script called mutt_oauth2.py to do the Oauth2 part. mutt calls this script to do the actual Duo multi-factor auth. We need to hardcode some values into the script (!) and then use it to initialize the OAuth token. Finding these values is an adventure in itself.

Unfortunately, in order to use this Python script we need GPG configured. For me this meant making a new GPG key.

For maximum confusion I will document these steps in (approximately) reverse order.

Configure GPG

tl;dr: Follow the instructions on the GnuPG ArchWiki page.

In particular, I set up a new key using gpg --gen-full-key

I set no expiry date (but this is stupid)
I set my name as "Paul Nijjar (UW Email)"
I set my email address as pnijjar.example.com
I set a passphrase, which I will not share.

I then needed to configure GPG to prompt me for passphrases appropriately. First, in ~/.bashrc I added the line export GPG_TTY=$(tty) . That is all I needed on the netbook.

On the Xubuntu laptop there is some popup that prompts me for a passphrase. I guess I am okay with this but it prompts me too often, and I do not want to store the passphrase in a keyring. Instead I want gpg-agent to cache it for a long time. The internet suggests that I make a file called ~/.gnupg/gpg-agent.conf with the following contents:

no-allow-external-cache
max-cache-ttl 86400
default-cache-ttl 86400

This will make GnuPG (in particular gpg-agent) remember a passphrase for 24 hours. Then to make this take effect I needed to kill (gpgconf --kill gpg-agent) and restart (gpg-agent) the agent.

Configure mutt_oauth2.py

tl;dr: Install the script. Customize the ENCRYPTION_PIPE setting, and then steal the necessary client ID and secret from Thunderbird's source code.

Download the mutt_oauth2.py script from the Mutt Gitlab page here: https://gitlab.com/muttmua/mutt/-/blob/master/contrib/mutt_oauth2.py .

Save the script someplace convenient. I put mine in my ~/bin folder, /home/pnijjar/bin/mutt_oauth2.py . Give the script user execute permissions: chmod u+x ~/bin/mutt_oauth2.py

There are two sections of the script in which we need to hardcode values. The first is to set the ENCRYPTION_PIPE variable with the email address you used when setting up your GPG key. In my case, I changed:

ENCRYPTION_PIPE = ['gpg', '--encrypt', '--recipient', 'YOUR_GPG_IDENTITY']

ENCRYPTION_PIPE = ['gpg', '--encrypt', '--recipient', 'pnijjar@example.com']

Next you need to specify the client_id and client_secret fields in the microsoft stanza of the registrations dict. Look for the following code:

    'microsoft': {
        'authorize_endpoint': 'https://login.microsoftonline.com/common/oauth2/v2.0/authorize',
        'devicecode_endpoint': 'https://login.microsoftonline.com/common/oauth2/v2.0/devicecode',
        'token_endpoint': 'https://login.microsoftonline.com/common/oauth2/v2.0/token',
        'redirect_uri': 'https://login.microsoftonline.com/common/oauth2/nativeclient',
        'tenant': 'common',
        'imap_endpoint': 'outlook.office365.com',
        'pop_endpoint': 'outlook.office365.com',
        'smtp_endpoint': 'smtp.office365.com',
        'sasl_method': 'XOAUTH2',
        'scope': ('offline_access https://outlook.office.com/IMAP.AccessAsUser.All '
                  'https://outlook.office.com/POP.AccessAsUser.All '
                  'https://outlook.office.com/SMTP.Send'),
        'client_id': '',
        'client_secret': '',
    },

In principle your Office365 administrator is supposed to generate a client ID and secret for you. IST eventually set me up with access permissions to do this, which I document in the next section.

In practice people just use the values that are hardcoded into the Thunderbird mail client source code. That seems like a terrible idea security-wise but it seems to work. As of this writing, that file contains the following stanza:

    "login.microsoftonline.com",
    [
      "08162f7c-0fd2-4200-a84a-f25a4db0b584", // Application (client) ID
      "TxRBilcHdC6WGBee]fs?QR:SJ8nI[g82", // @see App registrations | Certificates & secrets
      // https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-v2-protocols#endpoints
      "https://login.microsoftonline.com/common/oauth2/v2.0/authorize",
      "https://login.microsoftonline.com/common/oauth2/v2.0/token",
    ],

So the client_id and client_secret lines in mutt_oauth2.py become:

        'client_id': '08162f7c-0fd2-4200-a84a-f25a4db0b584',
        'client_secret': 'TxRBilcHdC6WGBee]fs?QR:SJ8nI[g82',

You will want to check the current values in the source code. These secrets do not live longer than 24 months, so depending on when you read this the actual client_secret is likely different.

Now we can initialize the OAuth2 token by calling the script: ~/bin/mutt_oauth2.py ~/.mutt_token --verbose --authorize

You should be asked for a registration (microsoft), a preferred OAuth2 flow (localhostauthcode), and an account e-mail address (your UW address, so pntestin@uwaterloo.ca for me).

If you are lucky you will see a giant URL you are supposed to copy and paste into a web browser. Paste that URL into your web browser and authenticate to UW using the Duo system. If you are lucky you will get a message like NOTICE: Obtained new access token followed by a large amount of random text representing the token you received.

If you are unlucky you will get an error message. I was getting AADSTS7000215: Invalid client secret provided. but that is because I had pasted the client_secret incorrectly and did not notice for hours.

Configure IST App Registration

tl;dr: The mutt_oauth2.py README is mostly correct, but as of this writing the platform is wrong.

I started out using the Thunderbird Client ID and secret, but I also asked the IST helpdesk whether we had something internal to UW. To my surprise, IST gave me access to the Microsoft Azure App Registration portal, and they (partially) set up a Client ID specifically for me. I needed to configure a Client Secret, and configure a bunch of permissions and settings. The above README had most of the information I needed, but (as of this writing) it appeared to have one error.

To complete this step it was very helpful to have two different web browsers running. I logged into Azure using Firefox, and ran the authentication using an incognito Chromium window. This way I could close the Chromium window and log in fresh again.

IST sent me an email telling me that I had been given access to the App Registration component: "PIM: You now have the Application Registration Owners role". Then I started configuring the app registration.

I logged into https://portal.azure.com . In the search bar I typed "App Registrations". This brought up the App Registration pages.
There was already an app registration set up for me called "pnijjar - mutt email application". This had a client ID, which I copied down to use in the mutt_oauth2.py script.

I then changed the following settings:

Authentication:
- Platform: "Web" (NOT "Mobile and desktop", as the README suggests)
- Redirect URL: "http://localhost"
- Front-channel Logout URL: Leave blank
- Implicit Grant and Hybrid Flows: I checked both "Access Tokens" and "ID Tokens" but I am not sure they are both needed.
- Supported Account Types: Accounts in any organizational directory (Any Azure AD directory - multitenant)
- Enable public client: off (the README says to enable this, but I didn't it and so far things seem okay?)
Certificates & secrets
- I created a client secret
- Name was arbitrary
- Set the expiry date to the maxmimum time (24 months)
- Copied the Client Secret Value (not ID). You need to copy this when you are creating the client secret -- you cannot see it afterwards.
API Permissions: (These were set for me already. If you need to set them, search for Microsoft Graph and then start typing the names.)
- Microsoft Graph, Delegated, "offline_access"
- Microsoft Graph, Delegated, "IMAP.AccessAsUser.All"
- Microsoft Graph, Delegated, "POP.AccessAsUser.All" (I actually removed this one, since I am using IMAP and not POP.)
- Microsoft Graph, Delegated, "SMTP.Send"
- Microsoft Graph, Delegated, "User.Read"

After saving this, I was able to use the new Client ID and Client Secret in mutt_oauth2.py and authorize. (I deleted the old .mutt_token file and started fresh each time I retried this.)

Explanations and Dead Ends

Why use the "Web" platform and not the "Mobile and desktop" one? When I tried the latter I got the error message AADSTS700025: Client is public so neither 'client_assertion' nor 'client_secret' should be presented. After a bunch of reading I learned that Microsoft classifies some apps as "confidential" and others as "public". "Web" apps are confidential (!) and "Mobile and desktop" ones are public. Public apps are more restricted in that they cannot pass around secrets, and client_secret is a secret.

Why specify a platform at all? I tried not doing this and received the error message AADSTS500113: No reply address is registered for the application.

Why specify the "Redirect URI" to localhost? It turns out that there is a redirect_uri parameter encoded in the giant URL you paste into Chromium. That was something like http://localhost:53841 where 53841 is a port that is randomly-generated by the script. I had other values there before; when I did so I got the error message AADSTS50011: The redirect URI 'http://localhost:53841/' specified in the request does not match the redirect URIs configured for the application I guess we got lucky that the Thunderbird app registration also uses localhost?

Why specify a multi-tenant application as opposed to a single tenant one? There are scary warnings ("Starting November 9th, 2020 end users will no longer be able to grant consent to newly registered multitenant apps without verified publishers") that show up if you specify multi-tenant. But when I specified single tenant I got the error message AADSTS50194: Application '...'(pnijjar - mutt email application) is not configured as a multi-tenant application. Usage of the /common endpoint is not supported for such applications created after '10/15/2018'. Use a tenant-specific endpoint or configure the application to be multi-tenant. I could not figure out what the tenant-specific endpoint was supposed to be. In the mutt_oauth2.py script we would need to change the tenant parameter to match this in addition to changing client_id and client_secret.

(Maybe) Recompile Mutt

tl;dr: Ignore this unless you are running a version of Ubuntu/Debian that has Mutt 1.x . Otherwise follow instructions from this defunct blog post.

The version of Mutt on Debian Bullseye was good enough that I could use it directly. The version of Mutt on Ubuntu Focal was too old. I ended up recompiling this, but it was not too difficult. My use case was simple: I just wanted a newer version, and there is one available in Ubuntu Impish.

First, add or change the deb-src line in /etc/apt/sources.list to point to Impish:

deb-src http://ca.archive.ubuntu.com/ubuntu impish main restricted

Next, run some commands to update your sources, install dependencies and make the .deb file. I made a folder called src for this. The commands that require root are prefixed with sudo:

mkdir src
cd src
sudo apt update
sudo apt-get build-dep mutt
apt-get -b source mutt

This downloaded the 2.0.5 version of Mutt and compiled it for me, resulting in a mutt_2.0.5-4.1build1_amd64.deb file. I could then install it as root:

sudo dpkg -i mutt_2.0.5-4.1build1_amd64.deb

I had installed other things like build-essential and devscripts earlier, so maybe you need to do that too? I am actively surprised that this compilation works, because the version of Mutt in Impish depends on something called libidn2 which does not exist in Ubuntu Focal at all. (There is an older library called libidn11 there.) Apparently this library is for internationalized domain names, which is no big deal for me but might hurt you.

Configure Mutt

tl;dr: None, really. You need this configuration to get mail working.

Finally you need to modify your .muttrc or equivalent. The relevant lines for my account were:

# Where should my inbox be?
set folder="imaps://pntestin@uwaterloo.ca@outlook.office365.com"
set imap_keepalive=15
set timeout=20

set imap_user="pntestin@uwaterloo.ca"
set spoolfile=+Inbox
set imap_authenticators="xoauth2"
set imap_oauth_refresh_command="/home/pnijjar/bin/mutt_oauth2.py /home/pnijjar/.mutt_token"
set smtp_authenticators = ${imap_authenticators}
set smtp_oauth_refresh_command = ${imap_oauth_refresh_command}

set smtp_url="smtp://pntestin@uwaterloo.ca@smtp.office365.com:587"
set ssl_starttls=yes

set from="pntestin@uwaterloo.ca"
set realname="Paul Nijjar (UW Testing)"
set hostname="uwaterloo.ca"

Note the weird format for the set folder and set smtp_url lines. They have two @ symbols each.

Note the imap_authenticators is set to xoauth2.

Note the imap_oauth_refresh_command points to the Python script and the token we authorized before.

I have other .muttrc settings (and you probably do too) but I think these are the ones I needed to get Mutt working with Duo.

Open Questions

I am not sure how often I need to reauthorize the token. Duo has an option to remember a token for 30 days but I don't want that. I also do not want to re-authenticate too frequently.

I do not know whether the Thunderbird client ID and client secret will work in the medium term. I do not know whether I will actually be able to generate a secret again in two years if I have been away from UW for several years.

I do not know what endpoint to use if I have a single-tenant application and not a multi-tenant one.

I do not know the correct setting for "Implicit Grant and Hybrid Flows" in the App Registration.

Obviously if there are security bugs in the version of Mutt packaged for Impish I will not get them unless I recompile and reinstall. I think that is unavoidable, but I am not sure.