Meeting setup

• Date: 26.03.2007
• Time: 09.00 Norw. time
• Place: Internet
• Tools: SubEthaEdit, iChat

Agenda

1. Opening, agenda review
2. Reviewing the task list from last week
3. Documentation - divvun.no
4. Corpus gathering
5. Corpus infrastructure
6. Infrastructure
7. Linguistics
8. name lexicon infrastructure
9. Spellers
10. Other issues
11. Summary, task lists
12. Closing

1. Opening, agenda review, participants

Opened at 09:48.

Present: Børre, Maaren, Saara, Sjur, Steinar, Thomas, Tomi, Trond

Absent: none

Agenda accepted as is.

2. Updated task status since last meeting

Børre

• update and fix our documentation and infrastructure as Steinar finds problem areas - low priority
  • nothing last week
• fix sme texts in corpus this month
  • not done
• find missing nob parallel texts in corpus
  • not done
• add prefixes to the PLX conversion
  • two of them
• add middle nouns to the PLX conversion
  • done
• improve number PLX conversion
  • not done
• add sma texts to the corpus repository
  • not done
• Improve automatic alignment process
  • not done
• Add potential speller test texts
  • not done
• collect a list of PR recipients, forward to Berit Karen Paulsen
  • not done
• add version info to the generated speller lexicons
  • not done
• run all known spelling errors in the corpus through the speller
  • not done
• test the typos.txt list, and check that all entries are properly corrected
  • not done
• consider how to do a regression self-test
  • not done
• fix bugs!

Maaren

• lexicalise actio compounds
  • Nickel s.310-315 (actio combounds)
• Manually mark speller test documents for typos
  • working with lundui.fi documents

Saara

• continue aligning the rest of the parallel files
• prepare more files for manual alignment
• mark-up the added speller test texts, using our existing xml format
  • done for the files that exist in prooftest
• improve cgi-bin scripts
  • add new features to the paradigm generator
    • not done
  • add gt/cwb/paradigm.smj.txt file into gt/script/server_anl.pl
    • done
• add new XSL/XML headers for proofing test docs
  • not done
• fix bugs!
• other
  • script for speller test data reformatting
    • first version ready
  • write conference paper
    • done

Sjur

• hire linguist
  • job interview done
• finish press release for the beta
  • not done
• collect a list of PR recipients
  • not done
• add version info to the generated speller lexicons
  • discussed with Polderland
• run all known spelling errors in the corpus through the speller
  • we’re not ready yet for that - depends on new feature in ccat
• consider how to do a regression self-test
  • no progress
• document the AppleScript testing tool
  • not finished
• write tools for statistical analysis of test results
  • started conversion to xml, handed the task over to Saara
• make improved smj speller (incl. derivations and compounds)
  • Tomi worked on this one
• fix bugs!
• other:
  • compiled new speller
  • other speller-related work
  • read up on OOo speller alternatives

Steinar

• Beta testing: Align manually (shorter texts)
• Manually mark speller test texts for typos (making them into gold standards), add the texts to the orig/catalogue
  • marked and tested some texts
• Complete the semantic sets in sme-dis.rle
  • no work this week
• missing lists
  • no work this week
• Look at the actio compound issue when adding from missing lists
  • not done
• Align corpus manually
• fix bugs!

Thomas

• work with compounding
  • worked and still working
• Lack of lowering before hyphen: Twol rewrite.
  • not done
• translate beta release docs to sme and smj
  • not done
• Add potential speller test texts
  • not done
• fix bugs!
  • not any this week

Tomi

• make improved smj speller (incl. derivations and compounds)
• add Makefile target for PLX conversion of lexc files
• make PLX conversion test sample; add conversion testing to the make file
• improve number PLX conversion
• update ccat to handle error/correction markup
• add version info to the generated speller lexicons
• fix bugs!

Trond

• Test the beta versions ** Done, added wds.
• Work on the parallel corpus issues
  • Nothing this week
  • Discuss with Anders
    • Nothing this week
  • Work on the aligner with (Børre)
    • Nothing this week
  • fix sme texts in corpus this month
  • find missing nob parallel texts in corpus, go through Saara’s list
• Postpone these tasks to after the beta:
  • update the smj proper noun lexicon, and refine the morphological analysis, cf. the propernoun-smj-lex.txt
  • Go through the Num bugs
• Improve automatic alignment process
• Align corpus manually
• Add potential speller test texts
• collect a list of PR recipients
• fix bugs!.

3. Documentation

The open documentation issues fall into these three categories:

• Beta documentation for testers
• Documentation for the online corpora
• General documentation improvement after Steinar’s test (for open-source release)

TODO:

• write form to request corpus user account (Børre, Sjur, Trond)
  • delayed till after the beta release
• document how to apply for access to closed corpus, and details on the corpus and its use in general (Børre, Sjur, Trond)
  • delayed till after the beta release
• correct and improve it based on feedback from Steinar (Børre)
  • low priority
• beta documentation (see separate beta section below)

4. Corpus gathering

Trond added one new text from Harald Gaski corp$ convert2xml.pl orig/sme/facta/file.doc

TODO:

• sme texts: no new additions, fix corpus errors during this month (Børre, Trond, Saara)
• missing nob parallel texts should be added if such holes are found (Børre, Trond)
• Go through the list of missing or errouneous nob texts, based upon Saara’s perfect list (Børre, Trond)
• add sma texts to the corpus repository (Børre)
• add missing words from new corpus text (Maaren)

  Go to victorio,
  cd gt/sme
  ccat -l sme zcorp/bound/sme/facta/Girjjalasv_doahpagat.doc.xml | preprocess
  --abbr=bin/abbr.txt --corr=src/typos.txt | lo | grep '\?' | cut -f1 | sort |
  uniq -c | sort -nr | less
  

5. Corpus infrastructure

Alignment

TODO

• go through other directories (nob dicrectories, sd directories), fix parallellity information for other documents (2 hours) (Børre)
• Improve the automatic process:
  • Improve the anchor list and realign (Trond, Børre)
  • Only adding words does not improve alignment, you have to consider the format as well. If you cut with star e.g guo* too early, wrong word can be selected.
  • The documents have still some formatting issues which cause trouble in alignment. (In some documents tables are included to the text, some not, etc.)
  • Test and improve settings in the aligner
• Align manually (Trond, Steinar) (especially shorter terminological texts) good idea. Saara will look for troublesome texts and prepare them for manual alignment.

6. Infrastructure

TODO:

• update and fix our documentation and infrastructure as Steinar finds problem areas (Børre)
  • started, working on it

7. Linguistics

North Sámi

Maaren is working on actio compounds, using the info found in Nickel. Thomas wants to make sure we’re understanding the issue in the same way. They will discuss this further after the meeting.

LEXICON BOAHTALADDAN ! not compounding
LEXICON BOAHTIN ! compounding, like: boahtin-

TODO:

• lexicalise actio compounds. Example: vuolggasadji vs. vuolginsadji (Maaren)
• fix stuorra-oslolaš lower case o (Sjur, Thomas, Trond)
  • postponed till after the public beta

Lule Sámi

TODO:

• refine smj proper noun lexica, cf. the propernoun-smj-lex.txt (Thomas, Trond)

8. Name lexicon infrastructure

Decisions made in Tromsø can be found in [this meeting memo.|/admin/physical_meetings/tromso-2006-08-propnoun.html]

TODO:

1. fix bugs in lexc2xml; add comments to the log element (Saara)
2. finish first version of the editing (Sjur)
3. test editing of the xml files. If ok, then: (Sjur, Thomas, Trond)
4. make terms-smX.xml <=== automatically from propernoun-sme-lex.xml (add nob as well) (the morphological section should be kept intact, in e.g. propernoun-sme-morph.txt) (Sjur, Saara)
5. convert propernoun-($lang)-lex.txt to a derived file from common xml files (Sjur, Tomi, Saara)
6. implement data synchronisation between risten.no and the cvs repo, and possibly other servers (ie the G5 as an alternative server to the public risten.no - it might be faster and better suited than the official one; also local installations could be treated the same way)
7. start to use the xml file as source file
8. clean terms-sme.xml such that all names have the correct tag for their use (e.g. @type=secondary) (Thomas, Maaren, linguists)
9. merge placenames which are errouneously in different entries: e.g. Helsinki, Helsingfors, Helsset (linguists)
10. publish the name lexicon on risten.no (Sjur)
11. add missing parallel names for placenames (linguists)
12. add informative links between first names like Niillas and Nils (linguists)

9. Spellers

OOo speller(s)

TODO after the MS Office Beta is delivered:

• add Aspell/Hunspell data generation to the lexc2xspell (Tomi - after the PLX data generation is finished)
• study Hunspell, perhaps also Soikko (Børre, Sjur, Tomi)
  • Aspell is not suitable
  • Soikko is not a viable alternative either (they are looking for replacing their technology with SFST - an interesting cooperation possibility in the future) - they’re using Malaga now (no HunSpell code).
  • HunSpell will be our choice for now

Testing

Selecting test texts

In principle, we need the same text types as the ones we already aim at in our corpus. In practice, we must use new, unseen texts. We would like to have a balanced input of texts, but right now:

• Min Áigi net issue
• Blogs
• Our own linguistic texts
• New department texts from the net

Storing test texts

TODO:

• Manually mark test texts for typos (making them into gold standards) (Steinar)
  • erorr§errors
  • when correcting to multiple strings: erorr.Og§(error. Og)
    • update correction markup xml conversion to handle the second case (Saara)
• Add the marked test texts to the corpus/prooftest/orig/ catalogue (Børre, Thomas, Trond, Steinar, Maaren)
  • done
• change ccat to handle error/correction markup (Tomi)
  • extract the document text with original errors (input to standard speller test) - default? yes
  • extract the document text with the available corrections (the correct docu), to be used to automatically evaluate testing tool output; could also be used to give the corpus tagger an easier job:-)
  • extract all and only the spelling errors with their corrections, in a tab separated file: erorr<TAB>correction - just as typos.txt. Used for regression testing (making sure we don’t take steps back)
  • extract both the original text and their corrections, in a tab separated file: erorr<TAB>correction - just as typos.txt. Used for another type of speller testing. correction should be an emtpy string if there is no correction.
• Set up ways of adding meta-information (source info, used in testing or not, added to lexicon or not)
  • add the wanted xml elements to the XSL header (Saara) (source info is the same as for regular docs):
    • outworned (ie not suitable for speller testing any more, only for regression testing)
    • lexicalised (all unknown, correctly spelled words added to lexicon)
• Conduct tests on new beta versions on the basis of the unspoiled gold standard documents (whoever has time), and fill in data from testing (the testers: who?)
• alternatively: make test scripts that will run the tests automatically, collect the numbers, and transform them into test results (Sjur)
• include the ones already tested in the testing/ catalogue
• test 0.3 on the same texts
• test each version before beta release

Testing tools

TODO:

• document the AppleScript testing tool (Sjur)
  • not finished
• write tools for statistical analysis of test results (Sjur)
  • started, Saara continued, will continue with Forrest integration

Regression tests

TODO:

• add extraction of all known spelling errors in the corpus (not the prooftest corpus), and check that they are properly corrected (Børre, Sjur)
• test the typos.txt list, and check that all entries are properly corrected (Børre, Sjur)
• consider how to do a regression self-test, ie, how to test the full wordlist (Børre, Sjur)

Localisation

We need to translate the info added to our front page (and a separate page) regarding the beta release. Also the press release needs to be translated.

TODO:

• translate beta release docs to sme (Thomas)
• translate beta release docs to smj (Thomas)

Lexicon conversion to the PLX format

TODO:

1. add gt/cwb/paradigm.smj.txt file into gt/script/server_anl.pl (Saara)
   1. done
2. add prefixes to the PLX (Børre)
3. middle nouns (Børre)
4. add Makefile target for PLX conversion of lexc files (Tomi):
   1. done
   2. adjectives
   3. nouns
   4. propernouns
   5. verbs derived into other POSes
   6. verbs - must be done on gtsvn.uit.no
      1. produced by the paradigm server on victorio? or regenerate every night, and make it available for regular download? (Tomi, Saara)
5. make conversion test sample; add conversion testing to the make file (Tomi)
6. improve number conversion (Børre, Tomi)
7. compile verb lists on victorio, make them available on our public server (Saara, Sjur)
8. speed issue with smj verb derivations - and this is only derivations to non-verbs (Tomi, Saara, Sjur)

Public Beta release

Due to the problems with generating the PLX files discussed above, we need to move the release date further. End of March?

Linguistic issues still open:

• prefixes (eahpe, ii-) (Børre)
• middle nouns (LEXICON: lexc: Rmiddle, plx: L) (Børre)

DONE:

• delivered PLX data of sme and smj including compounding
• translated Windows installer to sme and smj
• installed PLX compiler in G5 at /usr/local/bin/mklex* (one version for sme and one for smj)
• added resources needed for compiling PLX lexicons to our cvs repo
• tested the beta drop from Polderland - good we did, it is absolutely unacceptable (our responsibiliby - only linguistic errors (poor coverage) found so far)
• questions for Polderland:
  • version info in the speller
    • done
  • remaking/updating the installer packages with linguistic updates
    • we have received complete installer packages
• add compilation of MS Office spellers part of the Makefile
• install Windows and MS Office; test tools on Windows

TODO:

• improved smj speller (incl. derivations and compounds) (Sjur, Tomi)
• finish press release (Sjur)
• add info to front page (incl. download links) (Børre)
• write separate page with detailed info (incl. download links) (Børre)
• translate press release, web pages (Børre, Thomas, whoever)
• collect a list of PR recipients, forward to Berit Karen Paulsen (Børre, Sjur, Trond)

Version identification of speller lexicons

TODO:

• add version info to the generated speller lexicons (Børre, Sjur, Tomi)

10. Other

Project meeting IRL

The planned gathering will have to be on 16.-20.4., in Tromsø, most likely. All of Divvun should participate, probably some from the UiTø project as well.

Corpus contracts

TODO:

• publish corpus contracts and project infra as open-source on NoDaLi-sta (Sjur)
  • delayed until the public beta is out the door

Updates:

• MacOS: 10.4.9
• SubEthaEdit: 2.6.2

Bug fixing

48 open Divvun/Disamb bugs, and 23 risten.no bugs

Easter

• Børre: away
• Maaren:
• Saara: working
• Sjur: working
• Thomas: away 2/4 - 4/4
• Tomi: At work, in Karigasniemi
• Trond:

11. Next meeting, closing

The next meeting is 2.4.2007, 09:30 Norwegian time.

The meeting was closed at 10:49.

Appendix - task lists for the next week

Boerre

• update and fix our documentation and infrastructure as Steinar finds problem areas - low priority
• find missing nob parallel texts in corpus
• add prefixes to the PLX conversion
• improve number PLX conversion
• add sma texts to the corpus repository
• improve automatic alignment process
• add potential speller test texts
• collect a list of PR recipients, forward to Berit Karen Paulsen
• add version info to the generated speller lexicons
• run all known spelling errors in the corpus through the speller
• test the typos.txt list, and check that all entries are properly corrected
• consider how to do a regression self-test
• add extraction of all known spelling errors in the regular corpus (not the prooftest corpus), and check that they are properly corrected
• fix bugs!

Maaren

• lexicalise actio compounds
• Manually mark speller test documents for typos

Saara

• prepare more files for manual alignment
• mark-up the added speller test texts, using our existing xml format
• improve cgi-bin scripts
  • add new features to the paradigm generator
• add new XSL/XML headers for proofing test docs
• continue with speller test data
• compilation of verb lists
• speed in smj conversion
• fix bugs!

Sjur

• hire linguist
• finish press release for the beta
• collect a list of PR recipients
• add version info to the generated speller lexicons
• run all known spelling errors in the corpus through the speller
• consider how to do a regression self-test
• document the AppleScript testing tool
• write tools for statistical analysis of test results
• make improved smj speller (incl. derivations and compounds)
• fix bugs!

Steinar

• Beta testing: Align manually (shorter texts)
• Manually mark speller test texts for typos (making them into gold standards), add the texts to the orig/catalogue
• Complete the semantic sets in sme-dis.rle
• missing lists
• Look at the actio compound issue when adding from missing lists
• Align corpus manually
• fix bugs!

Thomas

• work with compounding
• Lack of lowering before hyphen: Twol rewrite.
• translate beta release docs to sme and smj
• Add potential speller test texts
• fix bugs!

Tomi

• make improved smj speller (incl. derivations and compounds)
• make PLX conversion test sample; add conversion testing to the make file
• improve number PLX conversion
• update ccat to handle error/correction markup
• add version info to the generated speller lexicons
• fix bugs!

Trond

• Test the beta versions
• Work on the parallel corpus issues
  • Discuss with Anders
  • Work on the aligner with (Børre)
  • fix sme texts in corpus this month
  • find missing nob parallel texts in corpus, go through Saara’s list
• Postpone these tasks to after the beta:
  • update the smj proper noun lexicon, and refine the morphological analysis, cf. the propernoun-smj-lex.txt
  • Go through the Num bugs
• Improve automatic alignment process
• Align corpus manually
• Add potential speller test texts
• collect a list of PR recipients
• fix bugs!.