Paul's Internet Landfill/ 2010/ Liberating Yahoo! E-mail with FreePOPs

Liberating Yahoo! E-mail with FreePOPs

As some of you know, my primary personal e-mail is hosted by Yahoo!. I am not happy with this situation (or how it came about), but I am sticking with it for now.

From 2007-2009, I accessed my Yahoo! mail via its web interface, using the text-based w3m browser. This was not great either, but it was usable -- at least until 2009, when Yahoo! changed its "classic" interface in a way that caused w3m to reload each page endlessly.

This change caused me pain, but it was probably a good thing. Using the web interface meant that I was storing all my mail on Yahoo!'s servers without a local backup. This w3m error motivated me to get that backup. (It also motivated me to start using mutt, which feels like hitting bottom. As long as I avoided mutt, I could tell myself that I wasn't one of Those People, and that I didn't really have a problem.)

Backing up webmail is a common problem. Many web businesses (including Yahoo!) make it much easier to add information to their services than they make it to get that information out again. Yahoo! Canada enables POP for its mail, so I was able to download my INBOX, but I also had several thousand messages in other folders, and I was too dumb to figure out how to configure POP access to those folders. In the worst case I could have moved all that mail back to my INBOX using the web interface, but because I was too dumb to figure out how to select all messages in a mailbox I would have been moving almost 9000 messages to my INBOX 200 at a time. Ugh.

Making my life easier was a message I learned from GMail: instead of organizing my mail into a deep taxonomy I would never use, I kept the number of folders I used minimal:

If I could figure out how to grab these two folders, I could back up my mail locally. Fortunately, a lot of people run into this problem. There exist several scripts that interact with the Yahoo! web interface to grab mail and make it available via a local POP daemon. Unfortunately, most of these scripts didn't work, because Yahoo! constantly modifies its web interface in ways that happen to break these download scripts.

As of November 2009, the solution that worked for me was FreePOPs . Here are some details about my experience.

Firstly, here was the environment I used:

Getting the FreePOPs service running was pretty easy. All of the pain occurs in the client configuration.

First, you have to set your client to use POP. (In Thunderbird I did this by making a new account.) The server name is localhost. The port is 2000, not the usual POP port of 110.

I configured Thunderbird to keep the mail on the account, and to use TLS if available. I also downloaded full messages, not just headers.

FreePOPs wants you to specify most of your configuration in the username field of your POP client. Here is a sample string based on the one I used:

example_user@yahoo.ca?folder=saved&view=All&keepmsgstatus=1

There are more options available. They are documented pretty well in the FreePOPs manual.

If you are lucky then you should be prompted by your Yahoo! password when you fetch mail for the account.

You have to change the folder= string in the username field for every folder you want to download, but Thunderbird/FreePOPs is smart enough to keep those folders separate in the local account. I don't know how to download nested folders, but it should be possible.

Because I had so many messages to download I was getting a lot of timeout errors. To fix this I increased the timeout in Thunderbird, as documented here: http://www.howinthetech.com/thunderbird-pop-timeout-problem/

The key is to edit prefs.js in Thunderbird: user_pref(“mail.pop3_response_timeout”, 300);

Even with this setting I was not able to download all my mail in one go. I would fetch my mail, get some messages, time out, wait a while and then fetch my mail again. On good runs I would get several hundred messages; on bad runs I would get eight. Fortunately Thunderbird (and FreePOPs?) are smart enough to continue downloading from the last download point even though we leave all mail on the server. It took me a couple of days of re-fetching my mail to get it all.

Apparently waiting for a while between runs might be a good idea. I remember reading something about Yahoo! not liking people who grab too much mail at once. (I think it is in the FetchYahoo! source code.)

Once the mail was downloaded I had to get it out of Thunderbird. This is easy because Thunderbird stores mail in standard (?) mbox format. The only tricky part is finding that mbox file. On Windows, it was hiding in a subdirectory of c:\documents and settings\<user>\Application Data\Thunderbird\Profiles . Once you have stored those files someplace safe, you have successfully liberated your mail.

I no longer use the Yahoo! webmail interface to compose mail. If you do, then maybe you need to keep around the Thunderbird account so you can periodically back up your mail again. But if possible I recommend not relying on FreePOPs to work forever, because (as far as I understand) it needs to continually play catch-up with changes to the Yahoo! webmail interface.

None of this addresses privacy concerns, of course. I assume that as soon as my mail touches Yahoo!'s servers Yahoo! will have a copy of that mail forever, regardless of whether I can access that copy or not. But I feel better having a copy of my own mail on my own computer, rather than trusting Yahoo! to safeguard it for me.