May 8
Searching for a new find
It’s time to start looking into a new search method for Thunderbird. One of the major changes planned for Thunderbird is a new and improve search, but what does that mean?
What do we have?
First lets look at what we have for a search system. At a very simple level most search systems break down into two pieces, a search interface for filtering and a results interface for listing. Thunderbird does this in a couple places.
Quick Search
The quick search entry is always at the top right of the Thunderbird window and allows people to search over the current view. The results of a quick search fill into the current view, replacing whatever listing was previously shown.
The Quick Search defaults to searching only the Subject or Sender and will only search mail that Thunderbird has downloaded already. Messages that are not listed in the current view (like in another folder) will not be searched unless that folder is selected, otherwise a person needs to use the Advanced Search.
Advanced Search
Hidden under the Edit Menu and Find Sub-Menu is an advanced search dialog that can make use of the remote mail or news protocol to perform a full search instead of just a local search. The Search Messages dialog provides it’s own search interface as well as it’s own results view directly below the search. While the Search Messages dialog provides some more advanced search methods over the quick search it’s hard to find and difficult to use effectively.
The Search Messages dialog allows for complex search queries to be built with multiple search terms composed of a number of different field type selectors. The queries require a lot of input from the user because of the tight structure used to create them. The same search and results interface code is used for creating mail filters.
Edit -> Find -> Search Messages… |
Advanced Search Dialog |
What do we want?
I was lucky enough to chat with Andrew Gilmartin yesterday and he framed a future goal very well. “We’re not looking to make search an added feature box on the side of Thunderbird“, we’re looking to make search the definitive method for viewing mail.
What does “Search as the definitive viewing method for your mail” mean? That’s a good question and I’m not sure exactly what a good answer is yet. A search would help you find the message you’re looking for, and perhaps a search view never lets you lose that message in the first place. There’s a lot to explore.
Here are two important pieces of a search system and view that need to be examined and somehow exposed in the interface.
Search and Filter
An impediment of the current search system is requiring people to choose a search type (Subject or Sender) before they even enter any text. To help people hunt for the correct item you want to allow for starting their search very broad and then allow them to narrow down that broad search with filters like subject or sender.
The current search system has some speed issues that likely prevented a broad to filter system of searching to be implemented. The mail client Mail.app provides a decent filter bar when searching mail that allows people to see what the current filters are (folder, account) and change them.
Browse and Filter
The SEEK extension is an excellent example of how offering a system of browsing mail by grouped attributes from the start can help people find the item or group of items they were looking for. Instead of starting with a search term you give the person a list of attributes they might use to filter the list of messages.
An inspiring system for a similar searching, browsing, and filtering methods is things, you should try it if you haven’t already.
Getting What we Want
Moving towards a new search based paradigm will take some adventurous steps and it’s important not to disturb current usage while making those steps. Here are a number of changes to look at making.
Merging Search Interfaces
Each of the two current search interfaces provide some needed features and capabilities, however having two separate interfaces for searching is confusing and difficult to understand. We need to combine the ability to do a quick search with the ability to perform a full search into a single interface with an improved results view.
With a single search interface Thunderbird will be searching the local and remote mail (like IMAP) at the same time. However local results will be listing quickly and remote results will likely take a little more time. Both sets of results, local and remote, can be merged into the same search results view by showing local results instantly and filling in remote results as they arrive.
Offline Cached and Indexed Mail
In order to have a fast search system even while offline Thunderbird needs to do a much better job of caching and indexing mail as it encounters it. With new messages instantly cached and indexed they can be made available to search queries, filters, and views immediately.
This is an excellent time to start thinking about the data mining mail in a way that helps searching messages later. It’s also time to think about making the defaults tuned towards offline usage while still allowing people to control online / offline caching.
Auto Complete
With mail data indexed locally and quickly available Thunderbird should be able to provide a slick and fun auto-complete on search terms it knows about. Auto complete when searching for items you’re already aware exists helps with miss-spelling errors and more complete matching. The awesomebar shows how with just a little broken memory of a title or url you can easily find the page you saw once before.
Fetching Results
Our current drive is to investigate some indexing on messages (at least subjects), pull the new auto-complete into Thunderbird, and get a search bar using that fancy auto-complete on mail subjects and hopefully the addition of a couple more fun things. Leave some comments or jump on the newsgroup to participate.
A wire frame of a possible mail search auto-complete
I have been strugling with cleaning my inboxes (work, school, personal) with Thunderbird. One thing that bothered me quite a bit was the inability to search all my folder simultaneously. As I am in a Mac, I then looked for a Spotlight plug-in, and found that the only one available is defective.
So here are my two cents: allow for searches across all folder, and enhance the support for OS specific searches (Beagle, Spotlight, what have you)
Whatever you do, please don’t break it, firefox already screwed up google search. Quick search as it is may not be the best, but it’s pretty useful and convenient.
I like the gmailUI addon approach to search.
http://www.longshot.com/~kmixter/gmailui.html
Marcus: There is someone in the Google SoC working on OS search integration for Vista right now; I believe Spotlight is what they are basing it off of.
Yevgen: No worries, the new search will be testing in the alpha and beta builds before it reaches anything critical. If it doesn’t perform as expected we’ll keep what we have and maybe just improve speed.
mish: I think we’re going to follow that kind of grammar for searching specific fields. I think we might even provide auto-complete for the grammar types so it’s faster and easier to use.
I would like search menu via the context menu.
Find related messages for the current message:
- from the same author
- with the same subject (ignoring Re:, Fwd:, …)
I think a big improvement for the Quick Search would be to make it global and find messages according to both location (name of the folder the message has been sorted into) and tags. Maybe add a checkbox or something that switches between global and local search.
Low hanging fruit: Please make “Match *all* of the following” the default.
Just a couple of comments. One of the things the current mail UI does best is being predictable. For instance, if you know what folder you email is in, you can directly navigate there. Any pure search based interface is lacking this predictability. So, please do not break what’s working already.
My current main issue with searching email is that I want the search find my most recent emails first, or at least have a choice from which end of my mailbox the search would commence. It’s highly probable that what I’m searching for is in the last 10 matching emails. When searching for an email right now, I need to think twice if I want to wait several minutes while a relevant result appears, or I’d be better of just manually scrolling through sorted messages in my mailbox.
Wow! I don’t remember even using the Advanced Search, but at the first sight the dialog looks familiar: is similar with the filter creation dialog, which is good enough for me (I rely heavily on filters).
But I use the Quick Search feature a lot, usually searching for subject and sender, because full body search would take quite a lot for some of my large folders (even if they are all local).
If the only option for Quick Search would be to search all folder, then full message search will become unusable for me.
[...] make an initial start on our new search aspirations we need to begin testing and trying out some of our improved searching [...]
I’ve created a Search Folder that lists all my messages (Edit > Find > Search Messages > Save as Search Folder > Match all messages).
I use the GMailUI extension mentioned above to search in this folder. I type all: foo bar into the quick search box (with “expression” selected) and every part of all my messages is searched for foo bar.
It’s slow and buggy, but at least I know that if what I’m searching for exists, it will be found – which I was never sure about with the default search options.
You might want to take a look at eudora’s search options.
The two things I miss the most in current TB ‘advanced’ search are relative simple options like:
– search ALL headers
not just from:, to: , cc:, etc, but *all* the headers in the messages, including Received:, X-… headers, etc etc
– search WHOLE message
Means all headers + whole body.. No way to do this right now.
I made the switch from eudora to TB a year ago and these operators not being present in advanced search still nag me. Have to resort to tools like grepmail on my mbox files, which is no fun when dealing with 40k+ message files..
Also options like “has attachment”, “attachment file name contains” , “Spam Score ” would be nice.. And being able to search for text *within* e-mail attachments (supportin most major text formats like plaintext, pdf, open office, word, etc etc) would really kick a**
Hello, please have a look at Evolution’s email Search Folders (search across your all mail folders with various criteria, right click the mail and there is shortcut to create this search folder).
I think this is really good idea.
Also take in mind TAGGING, it helps to quickly differentiate mail in Inbox and one can automatically have a search folder for that.
Mozilla rocks!
I wish there was a way to search the entire message in advanced search. Any time I have a lot of searches to do, I import my Thunderbird mail into Eudora and do the searches there. Perhaps the Eudora search engine could be used.