Meeting setup

Date: 30.04.2007
Time: 09.30 Norw. time
Place: Internet
Tools: SubEthaEdit, iChat

Agenda

Opening, agenda review
Reviewing the task list from last week
Documentation - divvun.no
Corpus gathering
Corpus infrastructure
Infrastructure
Linguistics
name lexicon infrastructure
Spellers
Other issues
Summary, task lists
Closing

1. Opening, agenda review, participants

Opened at 10:33.

Present: Børre, Sjur, Steinar, Thomas, Tomi, Trond

Absent: Maaren, Per-Eric, Saara

Agenda accepted as is.

2. Updated task status since last meeting

Børre

add sma texts to the corpus repository
- not done
collect a list of PR recipients, forward to Berit Karen Paulsen
- not done
run all known spelling errors in the prooftest corpus through the speller
- not done
add extraction of all known spelling errors in the regular corpus (not the prooftest corpus), and check that they are properly corrected
- not done
test speller installers on Windows and Mac
- we have a new multilingual Windows installer, which needs to be tested
  - the windows installer works without problems
update installer packages with latest speller lexicon
- trying to build installable spellers
add numbers, compound restrictions to both spellers if time permits
- not done
add numbers as figures to the PLX sources
- done
update and fix our documentation and infrastructure as Steinar finds problem areas - low priority
- not done
find missing nob parallel texts in corpus
- not done
Forrest dispatcher bug with spelltest results needs to be debugged
- made a workaround using xsltproc
study the Hunspell formalism in detail together with Sjur and Tomi
- Began reading [http://sourceforge.net/docman/display_doc.php?docid=29374&group_id=143754]
- Downloaded and compiled hunspell
test the typos.txt list, and check that all entries are properly corrected
- ran the make target that tests known typos. Result is available on the net /doc/proof/spell/testing/sme/mw/typos/20070425-typos.summary.html
Ask for larger disk for the webserver.
- done and installed
add info to front page (incl. download links)
- not done
write separate page with detailed info (incl. download links) (Børre)
- a separate page for the beta speller, with installation instructions, etc.
  - not done
translate press release, web pages (Børre, Thomas, whoever)
- not done
fix bugs!
other:
- fix bugzilla
  - bugzilla began working when we installed the new harddisk

Maaren

lexicalise actio compounds
Manually mark speller test documents for typos

Per-Eric

work on smj

Saara

prepare more files for manual alignment
improve cgi-bin scripts
- add new features to the paradigm generator
add new XSL/XML headers for proofing test docs
compilation of verb lists
read the manual for graphical corpus interface and try to add files with Lars.
fix bugs!

Sjur

finish press release for the beta
- not done
collect a list of PR recipients
- not done
run all known spelling errors in the corpus through the speller
- not done
finish documenting the AppleScript testing tool
- not done
integrate regression self tests with the make file
- not done
make improved smj speller (incl. derivations and compounds)
- tried a lot, several improvements to the make process, but sort still fails, although the last sort probably is ok (nohup added the stdout to the end of the file…)
improve speller test bench
- not done
update installer packages with latest speller lexicon
- not done
Forrest dispatcher bug with spelltest results needs to be debugged
- not done
integrate the ccat speller testing options in the make file
- not done
fix bugs!
other:
- tried to fix Per-Eric’s internet connection, to better support iChat, ARD and SEE.

Steinar

Beta testing: Align manually (shorter texts)
Manually mark speller test texts for typos (making them into gold standards), add the texts to the orig/catalogue
- working with it, so far over 10000 words from news
include the files already publically tested into the prooftest cataloge
- not done yet
Complete the semantic sets in sme-dis.rle
- no work lately
missing lists
no work lately
Look at the actio compound issue when adding from missing lists
- not done
Align corpus manually
fix bugs!

Thomas

work with compounding
- worked and still working
Lack of lowering before hyphen: Twol rewrite.
- not this week
translate beta release docs to sme and smj
- not done
Add potential speller test texts
- not done
fix bugs!
- had a look at some

Tomi

make improved smj speller (incl. derivations and compounds)
- Delegated to Polderland
add numbers, compound restrictions to both spellers if time permits
- not done
make PLX conversion test sample; add conversion testing to the make file
- not done
improve number PLX conversion
- not done
improve prefix and middle-noun PLX conversion
- not done
check whether the newest ccat is available in victorio/G5
- done
integrate the ccat speller testing options in the Makefile
- not done
fix bugs!

Trond

Test the beta versions
- Done, to the extent that they have been available…
Work on the parallel corpus issues
- fix sme texts in corpus this month
- find missing nob parallel texts in corpus, go through Saara’s list
  - No parallel work last week
Postpone these tasks to after the beta:
- update the smj proper noun lexicon, and refine the morphological analysis, cf. the propernoun-smj-lex.txt
- Go through the Num bugs
  - Closed some.
collect a list of PR recipients
- Not involved here. Sjur: not yet:)
Ask for larger disk for the webserver.
- Børre did this.
fix bugs!.
- Done a good job here, for once.

3. Documentation

The open documentation issues fall into these three categories:

Beta documentation for testers
Documentation for the online corpora
General documentation improvement after Steinar’s test (for open-source release)

TODO:

write form to request corpus user account (Børre, Sjur, Trond)
- delayed till after the beta release
document how to apply for access to closed corpus, and details on the corpus and its use in general (Børre, Sjur, Trond)
- delayed till after the beta release
correct and improve it based on feedback from Steinar (Børre)
- low priority
beta documentation (see separate beta section below)

4. Corpus gathering

Børre and Sjur had a small talk on Friday on how to continue the contract signing process, and thus get more texts.

TODO:

sme texts: no new additions, fix corpus errors during this month (Børre, Trond, Saara)
missing nob parallel texts should be added if such holes are found (Børre, Trond)
Go through the list of missing or errouneous nob texts, based upon Saara’s perfect list (Børre, Trond)
add sma texts to the corpus repository (Børre)

5. Corpus infrastructure

Alignment

The aligner is now working so well that we are not able to find errors anymore. We are now waiting for Lars to make new texts available, and the anchor list needs to be updated with new genres. Also, for the users, the error in the web interface made all texts seem misaligned (they were off by one sentence). This is also corrected now.

TODO

check with Lars when more texts are ready (Trond)

6. Infrastructure

TODO:

update and fix our documentation and infrastructure as Steinar finds problem areas (Børre)
fix internet setup for Per-Eric’s satelite modem (Sjur, Børre)
- this influences iChat, SEE sharing, and ARD connetions

7. Linguistics

North Sámi

TODO:

lexicalise actio compounds. Example: vuolggasadji vs. vuolginsadji (Maaren)
fix stuorra-oslolaš lower case o (Sjur, Thomas, Trond)
- postponed till after the public beta

Lule Sámi

The work is progressing very fine, a bit slow in the beginning, but that is expected.

TODO:

refine smj proper noun lexica, cf. the propernoun-smj-lex.txt (Thomas, Trond)

8. Name lexicon infrastructure

Decisions made in Tromsø can be found in [this meeting memo.|/admin/physical_meetings/tromso-2006-08-propnoun.html]

TODO:

fix bugs in lexc2xml; add comments to the log element (Saara)
finish first version of the editing (Sjur)
test editing of the xml files. If ok, then: (Sjur, Thomas, Trond)
make terms-smX.xml <=== automatically from propernoun-sme-lex.xml (add nob as well) (the morphological section should be kept intact, in e.g. propernoun-sme-morph.txt) (Sjur, Saara)
convert propernoun-($lang)-lex.txt to a derived file from common xml files (Sjur, Tomi, Saara)
implement data synchronisation between risten.no and the cvs repo, and possibly other servers (ie the G5 as an alternative server to the public risten.no - it might be faster and better suited than the official one; also local installations could be treated the same way)
start to use the xml file as source file
clean terms-sme.xml such that all names have the correct tag for their use (e.g. @type=secondary) (Thomas, Maaren, linguists)
merge placenames which are errouneously in different entries: e.g. Helsinki, Helsingfors, Helsset (linguists)
publish the name lexicon on risten.no (Sjur)
add missing parallel names for placenames (linguists)
add informative links between first names like Niillas and Nils (linguists)

9. Spellers

OOo speller(s)

TODO after the MS Office Beta is delivered:

add Hunspell data generation to the lexc2xspell (Tomi - after the PLX data generation is finished)
study the Hunspell formalism in detail (Børre, Sjur, Tomi)
- Børre did some reading, and compiled Hunspell

Testing

Spelling Error Markup

TODO:

Manually mark test texts for typos (making them into gold standards) (Steinar)
- continued the mark-up
Set up ways of adding meta-information (source info, used in testing or not, added to lexicon or not)
- add the wanted xml elements to the XSL header (Saara) (source info is the same as for regular docs):
  - worn out (ie not suitable for speller testing any more, only for regression testing)
  - lexicalised (all unknown, correctly spelled words added to lexicon)
Conduct tests on new beta versions on the basis of the unspoiled gold standard documents (whoever has time), and fill in data from testing (the testers: who?)
- we need new betas!
alternatively: make test scripts that will run the tests automatically, collect the numbers, and transform them into test results (Sjur)
- done, except that they don’t work with our tailored Forrest
  - Børre worked around this by using xsltproc to produce Forrest document format directly.
include the files already publically tested into the prooftest cataloge (Steinar, Trond)
add automatic testing to the make files for other test types (Børre, Sjur, Tomi)
test each version before beta release
- we don’t have a public beta release candidate yet:(

Testing tools

TODO:

document the AppleScript testing tool (Sjur)
improve speller test bench (Sjur)
- Forrest bug needs to be debugged (Børre, Sjur)
  - Børre found a work-around
- integrate the ccat speller testing options in the Makefile (Sjur, Tomi)
ask for Linux version of the Polderland command-line speller (for victorio) (Sjur)

Regression tests

TODO:

add extraction of all known spelling errors in the corpus (not the prooftest corpus), and check that they are properly corrected (Børre, Sjur)
- ccat now ready, it should be integrated in the Makefile (Sjur, Tomi)
test the typos.txt list, and check that all entries are properly corrected (Børre, Sjur)
consider how to do a regression self-test, ie, how to test the full wordlist (Børre, Sjur)
- extract all the base forms in the lexicon, and run them through the speller
- extract all SUB-marked entries, and run them through the lexicon
  - integrate these in the make file (Sjur)

Localisation

We need to translate the info added to our front page (and a separate page) regarding the beta release. Also the press release needs to be translated.

TODO:

translate beta release docs to sme (Thomas)
translate beta release docs to smj (Thomas)

Lexicon conversion to the PLX format

The conversion itself is working fine, but we are still having problems producing output acceptable to the mklex speller compiler. We have huge files to sort, and all sorts of problems pop up because of the size. We probably need new, big disks on victorio, and/or on the G5.

TODO:

ask MacOffice for larger disks for the G5 (Børre)
ask for larger disks for victorio (Børre, Trond)
ask for newer server OS on victorio, many of the installed tools are quite old (it is using Cent OS - a new version was released not long ago) (Børre, Trond)
ask for mklex for Linux (victorio) from Polderland (Sjur)

Compounding restrictions

How to include compounding restriction comment tags in the transducers:

giv0ri:giv'ri ALBMI ; !+SgNomCmp +SgGenCmp +PlGenCmp
=> (using a perl script or similar)
+SgNomCmp+SgGenCmp+PlGenCmpgiv0ri:giv'ri ALBMI ; !

TODO:

improve prefix conversion to PLX (Tomi)
improve middle noun conversion to PLX (Tomi)
improve noun + adjective PLX conversion: (Tomi)
1. compounding stems - how do we generate them? Using the java client? +SgNomCmp+Cmpnd = sáme–, should give the correct compounding stem, shouldn’t it? We want to optionally go from: sáme- NLI to sáme NL: - NLI (->) NL, which means we should be able to extract correct compounding stems using xfst methods only.
2. compounding tags - we need to obey them when making the transducers. Suggestion - see above.
make conversion test sample; add conversion testing to the make file (Tomi)
1. to regression test / QA the PLX conversion.
improve number conversion (Børre, Tomi)
ask for larger disk for the web server (Trond, Børre)

Public Beta release

TODO:

working and updated smj speller (Sjur, Tomi)
add numbers, compound restrictions to both spellers if time permits (Børre, Tomi)
finish press release (Sjur)
add info to front page (incl. download links) (Børre)
- download page made, only needs to add the speller beta when it is ready.
write separate page with detailed info (incl. download links) (Børre)
- a separate page for the beta speller, with installation instructions, etc.
translate press release, web pages (Børre, Thomas, whoever)
collect a list of PR recipients, forward to Berit Karen Paulsen (Børre, Sjur, Trond)
test speller installers on Windows and Mac (Børre)
- we have a new multilingual Windows installer, which needs to be tested
  - tested and ok
update installer packages with latest speller lexicon (Børre, Sjur)
- we need to test the procedure, and make sure it works

10. Other

Corpus contracts

TODO:

publish corpus contracts and project infra as open-source on NoDaLi-sta (Sjur)
- delayed until the public beta is out the door

Bug fixing

35 open Divvun/Disamb bugs, and 23 risten.no bugs

Sjur hasn’t received Bugzilla reminders this year… The reminder should include all non-closed/non-open bugs not touched in the last 14 days. Sjur, as a project manager, would like to have a status report on all open and untouched bugs.

TODO:

look over the Bugzilla status mails (Børre)

11. Next meeting, closing

The next meeting is 7.5.2007, 09:30 Norwegian time.

The meeting was closed at 11:30.

Appendix - task lists for the next week

Boerre

add sma texts to the corpus repository
collect a list of PR recipients, forward to Berit Karen Paulsen
run all known spelling errors in the prooftest corpus through the speller
add extraction of all known spelling errors in the regular corpus (not the prooftest corpus), and check that they are properly corrected
update installer packages with latest speller lexicon
add numbers, compound restrictions to both spellers if time permits
update and fix our documentation and infrastructure as Steinar finds problem areas - low priority
find missing nob parallel texts in corpus
study the Hunspell formalism in detail together with Sjur and Tomi
test the typos.txt list, and check that all entries are properly corrected
add info to front page (incl. download links)
write separate page with detailed info (incl. download links) (Børre)
- a separate page for the beta speller, with installation instructions, etc.
translate press release, web pages (Børre, Thomas, whoever)
update installer packages with latest speller lexicon (Børre, Sjur)
fix internet setup for Per-Eric’s satelite modem
ask MacOffice for larger disks for the G5
ask for larger disks for victorio
ask for newer server OS on victorio, many of the installed tools are quite old
fix bugs!

Maaren

lexicalise actio compounds
Manually mark speller test documents for typos

Per-Eric

expand the smj typos list
add missing smj words

Saara

prepare more files for manual alignment
improve cgi-bin scripts
- add new features to the paradigm generator
add new XSL/XML headers for proofing test docs
compilation of verb lists
read the manual for graphical corpus interface and try to add files with Lars.
fix bugs!

Sjur

finish press release for the beta
collect a list of PR recipients
run all known spelling errors in the corpus through the speller
document the AppleScript testing tool
integrate regression self tests with the make file
make improved smj speller (incl. derivations and compounds)
improve speller test bench
update installer packages with latest speller lexicon
integrate the ccat speller testing options in the make file
fix internet setup for Per-Eric’s satelite modem
ask for Linux version of the Polderland command-line speller (for victorio)
ask for mklex for Linux (victorio) from Polderland (for victorio)
look over the Bugzilla status mails
fix bugs!

Steinar

Beta testing: Align manually (shorter texts)
Manually mark speller test texts for typos (making them into gold standards), add the texts to the orig/catalogue
include the files already publically tested into the prooftest cataloge
Complete the semantic sets in sme-dis.rle
missing lists
Look at the actio compound issue when adding from missing lists
Align corpus manually
fix bugs!

Thomas

work with compounding
Lack of lowering before hyphen: Twol rewrite.
translate beta release docs to sme and smj
Add potential speller test texts
fix bugs!

Tomi

make improved smj speller (incl. derivations and compounds)
add numbers, compound restrictions to both spellers if time permits
make PLX conversion test sample; add conversion testing to the make file
improve number PLX conversion
improve prefix and middle-noun PLX conversion
integrate the ccat speller testing options in the Makefile
fix bugs!

Trond

Work on the parallel corpus issues
Postpone these tasks to after the beta:
- update the smj proper noun lexicon, and refine the morphological analysis, cf. the propernoun-smj-lex.txt
- Go through the Num bugs
collect a list of PR recipients
check with Lars when more aligned texts are ready
ask for larger disks for victorio
ask for newer server OS on victorio, many of the installed tools are quite old fix bugs!.