The MIME issue I had been experiencing was a result of coerceBodyToPlaintext() being incompatible with strictly “text/html” content types. asuth alluded to a bug in the function, but also noted that it only pertained to “multipart/mixed”, but it’s obviously more grave than that. A quick fix involved recursing on the MIME hierarchy one level deep, using the unaltered body when the part is text/html and appealing to coerceBodyToPlaintext() otherwise.

I also pushed a QuickLook patch, thanks to a bit of assistance from Daniel. We weren’t sure exactly how to account for whether or not QL was installed, but we think performing some exception handling on the “mail.qlxpcom.default.enable” pref was sufficient. It seems to handle gracefully in both cases.

Lastly, I did some last-minute buffing of code to make real-time pane refreshing tidier and more efficient (we’re not just emptying and then repopulating the pane when attributes are manipulated as before).

That’s about it. I’ve had a fun time hacking for Mozilla this summer and, while my contract officially has expired, there’s still a helluva lot of features to add before I’d feel comfortable slapping on a “1.0″, so you’ll hopefully continue to see changesets being added on a semi-regular basis to the repo. I’ll keep you posted!

At the moment, the HTML tests still fail because the pane refuses to display. asuth and I have had a back-and-forth regarding the issue and he reasons that it could be one of two things:

1) You are running your link detector against the ‘coerced’ fulltext body intended to be inserted into SQLite, not the actual html body. Since we ‘coerce’ HTML bodies to text bodies by using a HTML-to-text converter, perhaps ‘<a href=”foo”>bar</a>’ becomes ‘bar’ which is unlikely to trigger your regex.  If this is the case, you need to check out mimemsg.js and traverse the MIME object hierarchy yourself to find the pure html parts you want.

2) Our HTML support (or HTML coercion to text support) may be buggy. There’s a bug where it seems like that may be the case, although I think multipart/mixed may be the problem factor there (which is not involved with what you are doing).

The former point is an excellent segue into what I’ve done during the remainder of the week — that is, proper recursion of the MIME hierarchy — as can be seen in this patch. This handles 99% of my messages, except, you guessed it, the strictly HTML ones. These don’t occur terribly often as most messages containing HTML content are usually wrapped in a “multipart/alternative” object so as to allow for either means of displaying the message. Gloda seems to be baffled by these strictly HTML messages as it spews out debug statements that look like:

2010-08-09 15:03:07	gloda.NS	DEBUG	 found identity for '' (foo@bar.com)

The only call I make to coerceBodyToPlaintext() is if my MIME traversal yields an undefined body string to work with (so, basically, it’s just a form of error-avoidance in case I’ve overlooked some aspect of the MIME structure and, thus, this call shouldn’t ever be made in a perfect world). But, during the tests, my Attr code isn’t even executing in the first place, so that’s just wasted breath! Regarding point 2, I’ve just updated my cc build which yields no further changes to my situation.

In other news, tooltips have been added and there’s a serious proof of P != NP circulating… fascinating!

As mentioned in the previous post, I was having issues notifying my callbacks of Gloda’s completion. I managed to overcome this by scrapping the use of Gloda.isMessageIndexed() and instead using a GlodaIndexer listener to keep track of changes in Gloda’s status. I simply need to count the number of times Gloda’s status is set to kIndexerIdle and sleep on this variable (specifically, I wait until this count is 2 since the workflow is idle->indexing->idle). This is done on a per-folder basis.

The tests are divided into two files: plaintext message tests and HTML message tests. HTML source is only interesting in the case where we have hyperlinks as the regex parses these differently. The other files included in my ‘tests’ directory include a shell script for copying the tests into the mozmill directory and creating a temp file in the objdir containing the path to the dataminers extension as well as a patch for runtest.py which reads the temp file and copies the dataminers contents at this path into the test profile.

The tests are currently failing on:

• Certain tracking/phone number formats
• Messages containing both links and attachments
• Links occurring in HTML-formatted messages

• Modified CSS so that the image preview’s click functionality is distinguishable from the rest of the LI element’s.
• Fixed URL formatting bug where characters weren’t being parsed properly. The function encodeURIComponent() was indispensible in making sure URLs could be sent in a form POST request within a URL.

Wed., July 21

• Dynamic sidebar resizing when slideshow is started and closed. This is a simple (and possibly temporary solution) to the slideshow images being restricted, in size, to the boundary of the sidebar.
• I am awaiting a response from clarkbw regarding which sections checkboxes would enhance and which ones they would detract from. I am hoping the discussion can move towards this slideshow image issue.

Thurs., July 22

• I have been informed by asuth that the use of the following method in my tests was the reason for the lack of Gloda behaviour:

var addToFolder = function (msgId, aSource, aFolder) {
 var source = aSource;

 aFolder.QueryInterface(Ci.nsIMsgLocalMailFolder);
 aFolder.gettingNewMessages = true;
 aFolder.addMessage(source);
 aFolder.gettingNewMessages = false;

 return aFolder.msgDatabase.getMsgHdrForMessageID(msgId);
}

The call to addMessage() was not triggering message classification, so it was causing Gloda to appear inactive. asuth’s suggestion was to use functions similar to make_new_sets_in_folder(). My original hesitation was that this would not allow me to specify my own attachments/body/etc., but upon inspecting makeMessage() and functions calling it, I discovered I had complete freedom in message generation. My fallacy was to attribute the fault to the Mozmill process rather than my use of the message generation functions.

Fri., July 23

• More tests! I am currently trying to work out an elegant means of waiting for Gloda to finish indexing before simulating clicks on test emails. My approach involves sleeping on Gloda.isMessageIndexed(), but this seems to loop infinitely…

• Plugged in Fancybox to display individual images

Tues., July 13

• Wrote my evaluation
• Broke “Attachments” and “Images” into two separate sections

Wed., July 14

• Added slideshow which supports galleries of images
• Actions for attachments/images now displayed in a menupopup

Thurs., July 15

• Added resolution for images
• Tweaked CSS

• Had to scrap the use of the mozmill extension for testing as fake mail accounts had to be done at a lower level. The tests now reside in my comm-central repository under mail/test/mozmill.

Tues., July 6

• Hacked runtest.py located in my objdir’s mozmill test directory. I modified it to read a plaintext file called ‘addons’, where each line is a path to the desired addon. The contents of each specified addon are copied to the test’s profile directory. This allows for custom extentions to be enabled when ‘make mozmill’ is called in the objdir. Not the most elegant method, but it works!

Wed., July 7

• Played with message source to manually create attachments in a local folder.
• Though testPaneDisplayed() (which detects the new sidebar) is passing, the pane does not appear to be displayed. Gloda indexing is enabled and AFAIK, local folders are indexed. There must be some other prefs being specified which are causing the suppression of Gloda hooks.

Thurs., July 8

• Unfortunately, I had to put unit tests on hiatus due to the fact that Gloda was not indexing local test messages even after toggling ‘mailnews.database.global.indexer.enabled’ in runtest.py. I have been instructed to ask asuth about it next week.
• In the meantime, I’ve been adding a feature in the dataminer preferences which allows the user to toggle the social networking services he/she is interested in.

Fri., July 9

• Added aforementioned preference.

Sat., July 10

• Started re-vamping the attachment preview.

Mon., June 28

• Addressed Blake’s recommended changes as per code review. Mostly large-scale refactoring was involved.
• Blake suggested writing up some mozmill tests, which had been on the back of my mind for a while.

Tues., June 29

• Finished refactoring.
• Worked on two bugs: 1) In certain MIMEtypes, URLs are extracted as HTML-safe strings. However, these are further passed through scanTXT in the sidebar layer (e.g. &lt becomes &amplt). 2) Debugged attachment-deletion issue.

Wed., June 30

• Began learning/writing mozmill tests.

Mon., June 21

• Determined cause of hangs: inefficient regex matching in link_attr.js
• Made a judgement call and restricted the regex to only look for URLs for the time being — will have to discuss the issue with Blake further
• Fixed the phone number section display to account for many-to-many relationships between people and phone numbers
• Fixed a tracking number bug where only one number is displayed per message

Tues., June 22

• Upon pushing to https://bitbucket.org/ilienert/dataminers, Blake informed me that I had deleted some changes made in the last revision. What ended up happening was that I had been working off a previous revision and neglected to notice the changes to phone-related code. In fear of a slew of conflicts ensuing, I decided to manually incorporate the changes.
• Due to copyright concerns, I removed the Web 2.0 icons I had been using. Next on the list is to hunt down some suitable-for-commercial-use ones.
• Added a quick blurb to the MDC to direct new devs to some documentation for formatting of query strings — a problem I had been stumped by for a couple of days.

Wed., June 23

• Incorporated changes from commit 51:f09c560bbe86 manually.
• Fixed checkbox bug. The cause of the problem was traced to two sources. First, the jQuery click() function was not bubbling up the DOM due to return statements inside the handlers. Secondly, the toggle() function was applied directly to the checkbox input element which also resulted in default behaviour being overridden. To fix this, I replaced toggle with the convenient change() function.

Thurs., June 24

• Rounded up social media icons (free for commercial use)

Fri., June 25

• Added “Add phone number to > (Home | Work | Cell | etc.)” feature

Sat., June 26

• Bugfix: When bulk actions were performed (e.g. adding a selection of contacts to the address book), the content of the pane would be generated multiple times. This was a result of the address book listener picking up on individual changes made during a batch operation. So, each time a contact card was modified, onMessageDisplayed() was called. To resolve this, I simply had to suspend the listener for the duration of the operation.
• Exploring a issue wherein the pane is permanently cleared upon detaching attachments — likely a MIME issue.

Next Steps

• Hoping to hear from Blake re: my first deliverable. Likely, he’ll have a number of suggestions to keep me busy for the coming week.
• At Blake’s discretion, I’d like to tackle phase 2 of the project — the slideshow. I’m eyeballing jQuery’s Fancybox and, if possible, I’d like to integrate Andrew Valum’s nifty scroll menu.

(Unable to contribute due to graduation)

Tues., June 15

• Added checkboxes for attachments
• Save/delete/detach selected

Wed., June 16

• Added checkboxes for persons
• Add/remove selected
• Started playing around with mozrepl which affords editing chrome from within Tb and greatly reduces the overhead of debugging

Thurs., June 17

• Added remaining checkboxes
• Rebuilt Tb with –enable-logrefcnt and –enable-trace-malloc and started playing with Valgrind so as to track down the source of some occasional memory leaks/bloat

Fri., June 18

• Mostly documentation reading
• Trying to understand the formatting of query strings appended to URI’s (e.g. as seen in https://developer.mozilla.org/En/Address_Book_Examples#How_do_I_search_for_multiple_contacts.3f)
• Determining source of regression which seems to restrict onclick callbacks from executing their scripts

Sun., June 20

• Refactored/cleaned up code
• Addressing bug where pane is not refreshed after Gloda has indexed the message

Mon., June 7

(I was unavailable to work on my project this day. I was attending a funeral for a family member.)

Tues., June 8

• Started “Show Contact Info” for persons added to the database
• Stumped by the address book URI (have to pass in abURI as an argument to the Edit Contact dialog)
• Currently using “moz-abmdbdirectory://abook.mab” which does not result in any JS errors in the console, however I am not able to modify card details

Wed., June 9

• Solved former problem by replacing “moz-abmdbdirectory://abook.mab” with “moz-abmdbdirectory://history.mab” which refers to “Collected Addresses”
• “Remove Contact” feature
• Sidebar is automatically updated when a person is added/removed
• Need to disable the “Email” textbox when the “Edit Contact Card” dialog is invoked, otherwise a new contact will be created with that address and the sidebar will henceforth bring up this card when “Show contact info” is clicked

Thurs., June 10

• Disabled “Email” textbox by overlaying abEditCardDialog.xul; Had to pass an emailDisabled argument to openDialog so that my overlay could determine whether it should disable the field (dataminers) or not (address book)
• Starting delving into three bugs/features

Certain messages are not being mined properly — likely a long-term bug

Duplicate contacts appear when attachments are detached/deleted

Close button on attachment panel/closing the panel when buttons on it are clicked (serious: when the “Save As” window is brought up, it is obscured by the panel, but the window hijacks thread priority!)

Fri., June 11

• Attachment panel working perfectly
• Duplicate problem fixed by checking for duplicates in addPersons (haven’t nipped this one in the bud, but testing such an approach would require a lot of reindexing — *barf*)
• Attachment pane now refreshes whenever a change is made to phone number-identity relations
• Started looking into XPCNativeWrappers to avoid streaming image links into the attachment pane prematurely

Next Steps