Typo Pad

Searching for Functional Words

Re: Desperately Seeking Public Mailing Lists

Because the LinkWad link I posted earlier is not very self-documenting, here is a brief attempt at explaining what is going on.

I started out by collecting small functional words for the languages I am interested in at this point.  For languages which I don't speak at all, the Wikipedia pages in that language were useful for picking up some apparent functional words (for example, Albanian and Catalan got this treatment).

I want functional words which are frequently used and ideally which don't easily clash with those in other languages, although that is of course very hard to guarantee.  At the very least, it would be good if I have distinctive words for closely related languages, so that I don't get them mixed up.

Finally, in order to avoid misses or other accidents due to problematic, erroneous, or nonstandard encodings, I wish to use only words which are pure 7-bit ASCII.  This turned out to be problematic e.g. for Albanian, where most of the candidate words have accents.

To these functional words, I added some search terms in order to focus the search on mailing lists.  Specifically, I want "Subject" and "Message-Id", as in email headers, in the hits.  Moreover, I added inurl:pipermail in order to require that the hits have the word "pipermail" in their URL; as documented earlier, I try to make this as simple and standard as possible for myself at this point, so I want hits from a widely used standard mailing list archive manager which uses a simple, downloadable text-only format (Berkeley mbox files).

Here are the resulting searches.  I only list the functional words I picked for a language;

• English: there the with who
• German: aus bei sind zu
• French: mais sont les elle dont
• Italian: su sei della nel
• Spanish: el que y las
• Catalan: ens us fer i
• Portuguese: um uma em na
• Swedish: av från på med improved av till och ett finds more lists too, 783 hits -- apparently precisely because of character encoding problems.  I should have known!
• Danish: af fra til (should have one or two more?)
• Finnish: oliko mutta saa voi (not very functional,  mostly verbs -- update?) improved ei joka on mutta (246 hits)
• Estonian: siit mis mida ka
• Albanian: dhe nga me tij

I also tried to peruse the Icelandic and Faroese Wikipedias in order to come up with a good list of minimal word pairs which would distinguish them from each other, but I failed -- the results I got for my Faroese search also included some Icelandic, and apparently some Old Norse as well.  Besides, there don't seem to be any Pipermail archives in Google with these search words in them.

I am also noting here for possible future use:

• Icelandic: sem eru um af (106 hits)
• Polish: w do tej od z (763 hits)
• Letzebuergisch: vu vun vum zu op der (8 hits) -- I had to add more words because "vun" seems to be popular in some base64 attachments ...
• Afrikaans: by deur oor sy (39 hits)
• Basque: eta bat ez da bi (319 hits)

For languages with their own writing systems, I imagine I will need to take a different approach.  First of all, I'll need to establish what the possible encodings are, and perhaps roughly their relative ratios of real-world use.  Perhaps then it would be enough to collect material in UTF8 and use recode to convert to the other possible formats.  And that probably still won't cut it for languages with a Cyrillic writing system.

For the longer term, Wikipedia currently has pages in 250+ languages.  One could assume that there will be a rough correspondence between Wikipedia activity in a language and the availability of public mailing list archives in that language.

Exercise: find search words for Greenlandic -- it's a heavily agglutinating language, so there aren't any small particles to latch to.  (Actually Navajo appears to be even harder, not least because it seems impossible to type on a standard keyboard, but then, let's not even get into various exotic ideographic scripts.)

Desperately Seeking Public Mailing Lists

Re: Multilingual Email Corpus Project

I saved some canned searches to a public LinkWad -- results right now are as follows:

• English -- 109,000 hits
• Swedish -- 489 hits
• Danish -- 457 hits, perhaps too wide search
• Finnish -- 39 hits, rather little, too narrow search?
• Estonian -- 7 hits, clueless selection of search words? ("omitted results included" yields ... 8)
• German -- 5,810 hits
• French -- 1,220 hits
• Italian -- 2,480 hits
• Portuguese -- 7,710 hits, perhaps not the best search words

Update: I also think I have these covered now, in spite of what I thought earlier;

• Spanish -- 11,900 hits
• Catalan -- 620 hits

In the worst case, I will simply have to exclude Finnish and Estonian from this phase.  The idea is to focus on the really low-hanging fruit, although I also want Far East languages in the set for ... particular reasons.  I'll need to get back to those as well.

For now, I'll probably work exclusively on Pipermail archives, because they are fairly common and structured in a standard way; thus, they should be easy to collect.  Maybe later I'll want to expand the toolchain to cope with other types of archives (Mhonarc? Mailman? Listserv? Yahoo Groups? Google Groups (and NNTP!)? Gmane?)

I want a fairly broad selection of lists in order to avoid the obvious biases -- many messages from the same people, or on the same (probably computing / Internet / open source-related) topics, and of course it would be nice if there were different levels of formality and familiarity in the corpus, but that might be harder to achieve with public, open mailing lists as the sole source.

Multilingual Email Corpus project (January commitment)

As indicated in my initial posting, the plan for this blog was mainly to announce to the world what I'm up to, in order to attain some focus and perhaps some peer pressure from my $dmr-zillion alert and supportive readers.

To my utter surprise, signing up for this 14-day trial account and familiarizing $self with Typepad has already consumed the better part of my Saturday.  Maybe I can book this time under "cost of doing business" but seriously, now would be a good time for some officious declarations.

I obtained my Bachelor's degree in 2000, and have been meaning to finish a Master's before I turn 40.  I have for six and a half years had an agreement with the benevolent current employer that time would be allocated for this "at some point, as projects permit".  In case you are not entirely up to speed with corporate BS, that means "never".  But now, the old alma mater is exerting some pressure, because they are in the final phases of a transition to a completely new exam structure; and so, they are demanding that you finish your degree now (concretely, before June) or deal with the consequences.  In the worst case, we are told, those might include the abyss of a freshman course in using computers for web surfing, email, and word processing.  And in fact, April would be nice, they say, because they expect quite the real-world equivalent of a Slashdot phenomenon as hordes of old $bleeps (attempt to) get their acts together.

OK, enough background.  My commitments for January are:

• Finish the synopsis for my thesis which I have on this very laptop, freshly restored from the old dying hard drive; and
• Start compiling a multilingual email corpus from public mailing list archives

The corpus is just a sidetrack from the real thesis work, but it's a prerequisite for the actual research I make believe I will perform.  I even have some age-old notes about how this might have something to do with spam filtering, but the way things look now, the spam topic will not really be the focus of the thesis. (Incidentally, note to self; I should also look at this stuff, especially the enigmatic reference to "Kranig".)

Because I cannot reliably identify all the, what, 47? languages in TextCat, I will have to start with a smaller selection, but I believe I should be able to cope with at least the following:

• English
• Swedish
• Danish
• Finnish
• Estonian
• German
• French
• Italian
• Portuguese
• Japanese
• Korean
• Chinese

And if I have the time, and find suitable mailing lists:

• Polish
• Hungarian
• Romanian
  
• Albanian
• Turkish
  
• Greek
• Basque
  
• Esperanto
• Vietnamese

(The majority of these are easy because they are easy to identify from the word shapes or the writing system used, not because I am a language wonderchild.  Quite frankly, as a linguist, I am ashamed that I am not able to handle more than the three and a half languages which were compulsory at school.)

Slightly more problematic:

• Norwegian generic vs Nynorsk vs Bokmål?
• Dutch vs Afrikaans?
• Icelandic vs Færøese -- probably not hard at all with a bit of background study
• Latvian vs Lithuanian -- ditto
• Spanish - Catalan vs Castellan -- ditto
• Hebrew vs Yiddish -- ditto
• Czech vs Slovak?  Probably easy to mix up with some other Slavic languages too
• Russian vs Ukrainian vs other languages with Cyrillic writing, some of which I have never heard of?
• Arabic vs Farsi vs other languages which use Arabic writing

Some of these are problematic only because I'm ignorant of the differences, but that doesn't help.  I don't think I could tackle many more even if my legions of readers were able to help with some of these.  And I apologize already in advance to everyone whose favorite language is not on any list.

It would be fun to add something like Dzongkha if I can find adequate resources, but maybe that's too optimistic ...

I'll try to post a follow-up soon with some method notes.  (Update: here.)