Meeting setup

Date: 21.8.2007
Time: 10.30 Norw. time
Place: Internet
Tools: SubEthaEdit, iChat/Skype

Agenda

Opening, agenda review
Reviewing the task list from last week
Documentation - divvun.no
Corpus gathering
Corpus infrastructure
Infrastructure
Linguistics
name lexicon infrastructure
Spellers
Other issues
Summary, task lists
Closing

1. Opening, agenda review, participants

Opened at 11:14.

Present: Børre, Ilona (last part), Maaren, Per-Eric, Sjur, Thomas, Tomi, Trond

Absent: none

Agenda accepted as is.

2. Updated task status since last meeting

Børre

add sma texts to the corpus repository
- Not done
run all known spelling errors in the prooftest corpus through the speller
- Not done
add extraction of all known spelling errors in the regular corpus (not the prooftest corpus), and check that they are properly corrected
- Not done
update and fix our documentation and infrastructure
- Not done
follow-up contact with Davvi Girji
- Done. Michal Aase had just arrived from vacation, will send us texts when he finds time.
fix bound corpus access for external users (documentation, application)
- Not done
move error markup in xml files to (a copy of) the originals
- Not done
fix bugs!
Other:
- Continued working with sfst.

Maaren

lexicalise actio compounds
- worked and still working

Per-Eric

expand the smj typos list
- worked and still working
add missing smj words
- worked and still working
fix errors in smj/src/typos.txt
- done
lexicalise words from the Olavi missing list
- worked and still working
test new speller for actios of 3-sybbable verbs, adverbs of 3-s adjs and æ/ä
- done some
add compounding tags to adjectives
- done allmost ready
ask SD/Sweden about smj place names
- done
call Kåre Tjikkom about smj place names and correct-doc
- done
call Samuel Gælok about texts
- done

Saara

add new XSL/XML headers for proofing test docs
Try to add files with Lars to the corpus interface.
fix bugs!

Sjur

speller testing:
- run all known spelling errors in the corpus through the speller
- document the AppleScript testing tool
- integrate regression self tests with the make file
  - tested the baseform extraction tool made by Tomi
- improve speller test bench
- integrate the ccat speller testing options in the make file
fix internet setup for Per-Eric’s satelite modem
- nothing done
check with Sámi publishing houses whether support for CS2 is still needed
- only Iđut left - tried to call, no answer. Board decided we will drop it anyway (I don’t think it is a big problem for Iđut).
publish corpus contracts and project infra as open-source on NoDaLi-sta
- nothing done
fix bound corpus access for external users (documentation, application)
- nothing done
fix stuorra-oslolaš lower case o
- nothing done
ä/æ in smj speller
- nothing done
restart work on the XML name editor/risten.no integration
- done some details
plan the rest of the project period
- nothing done
fix bugs!

Thomas

work with compounding
- worked
fix stuorra-oslolaš lower case o
- not done
test new speller for æ/ä
- not done
fix bugs!
- not worked

Tomi

make PLX conversion test sample; add conversion testing to the make file
- not done
add Hunspell data generation/conversion
- not done
fix bugs!

Trond

Work on the web corpus issues
- Not done.
update the smj proper noun lexicon, and refine the morphological analysis, cf. the propernoun-smj-lex.txt
- Not done.
fix bound corpus access for external users (documentation, application)
- Not done. Oslo will administer the external access. Docu not written.
fix stuorra-oslolaš lower case o
- Not looked into.
fix bugs!.
- Worked on bugs, fixed many.

3. Documentation

Corpus web interface: Oslo will take care of everything. Unix access: we’ll bring it up if needed. For the time being web access should suffice for most users.

4. Corpus gathering

Per-Eric didn’t meet with Sigga Tuolja-Sandström, but talked with her instead. The text licensing contracts should be fixed now. Per-Eric will get the text from another person.

Børre talked with Michal Aase, and they (Davvi Girji) will send us all texts for which we have a contract with the authors.

Per-Eric talked with Samuel Gælok, who said that Iđut has several of his texts. There’s also another person/publisher in Karasjok that has some of his texts.

Kulturminnelova in smj is only available in paper form, unless Anders Kintel has an electronic copy.

TODO:

add sma texts to the corpus repository (Trond)

5. Corpus infrastructure

Nothing this week either.

6. Infrastructure

Børre got the G5 working again. It was caused by a misconfiguration in the firewall.

We are ordering a new server for faster processing.

7. Linguistics

North Sámi

Remaining twol issues:

###  Changed because:we get olmmož-, not olmmoš-

olmmošmuorra
olmmošmuorra    olmmošmuorra    +?

olmmožmuorra
olmmožmuorra    olmmoš+N+SgNomCmp#muorra+N+Sg+Nom

###  Changed because:we get almmáj- and not almmái-
 but this works now:
 almmájmuorra
almmájmuorra    almmájmuorra    +?

almmáimuorra
almmáimuorra    almmái+N+SgNomCmp#muorra+N+Sg+Nom

almmáj-
almmáj- almmáj- +?

almmái-
almmái- almmái+N+SgNomCmp+Cmpnd

We are now back to two errors in the twol-test file:

### € olmmožX4X7-
### € olmmoš00-
  #:0
###
  €
  o
  l
  m
  m
  o
  ž:š
  X4:0
  X7:0
  REJECTED: "Word Final Consonant Neutralization 1" fails in state 17.

### € hálijd#
### € háliit0
  #:0
###
  €
  h
  á
  l
  i
  j:i
  d:t
  REJECTED: "Word Final Consonant Neutralization 1" fails in state 27.

The sme names from Finland is still not added to the lexicon (Børre has them). This will be done by Ilona.

TODO:

lexicalise actio compounds. Example: vuolggasadji vs. vuolginsadji (Maaren)
fix stuorra-oslolaš lower case o (Sjur, Thomas, Trond)
add the sme place names from Finland (Ilona)

Lule Sámi

The æ-ä alternation issue has turned into an interesting direction. With the latest speller (29.6.), it behaves like the following:

dæbbaga -  ok
däbbaga -  däbboga
           dæbbaga
           dæbbaga--
           dibága
           dibága--
           dubága
           dubága--
           tubága
vællahit - ok
vällahit - vællahit
           vellahit
gæhtjáj  - ok
gähtjáj  - ok

That is, ä works in some cases, but not in other. æ seems to work everywhere it should.

Per-Eric talked to Nils-Olof Sortelius at Sámediggi in Sweden (smj place names), we will have to contact Lennart Dehlin, Lantmäteriverket Lennart.Dehlin(at)lm.se. Also talked to Kåre Tjikkom, who gave P-E the name of the SD/N contact person in place name issues: Lisa Monika Aslaksen.

Kåre Tjikkom has lost his speller-correct document. Sjur will have to find it, or make a new one.

This is probably a speller issue, not a twolc one. The transducers work fine:

däbbaga
däbbaga deppa+N+Sg+Gen
däbbaga deppa+N+Pl+Nom
däbbaga dæppa+N+Sg+Gen
däbbaga dæppa+N+Pl+Nom

vällahit
vällahit        vællahit+V+IV+Inf
vällahit        vællahit+V+IV+Imprt+Pl2

gähtjáj
gähtjáj giehtje+N+Sg+Ill

From invert transducers (regular and normative):

Tomi-si-maskin:~/Documents/eclipse/workspace/gt tomi$ lookup -flags mbTT -utf8 smj/bin/ismj.fst
0%>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>100%
dæppa+N+Sg+Gen
dæppa+N+Sg+Gen  dæbbaga
dæppa+N+Sg+Gen  däbbaga

Tomi-si-maskin:~/Documents/eclipse/workspace/gt tomi$ lookup -flags mbTT -utf8 smj/bin/ismj-norm.fst
0%>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>100%
dæppa+N+Sg+Gen
dæppa+N+Sg+Gen  dæbbaga

Normative produces word only with æ.

TODO:

refine smj proper noun lexica, cf. the propernoun-smj-lex.txt (Thomas, Trond)
ä/æ in speller (Thomas, Sjur, Trond)
- does only work partially, see data above.
lexicalise words from the Olavi missing list, but check against the pdf original where in doubt (Per-Eric)
- needs help with the last part
add compounding tags to (some weeks of work):
- nouns (Thomas)
  - a few weeks left
- adjs (Per-Eric)
  - done most of them, will have to recheck some
ask SD/Sweden about smj place names (Per-Eric)
- done
call Samuel Gælok about texts (Per-Eric)
- done
call Kåre Tjikkom about smj place names and correct-doc (Per-Eric)
- done
resend smj speller-correct document to Kåre Tjikkom (Sjur)

8. Name lexicon infrastructure

This sub-project needs to get up and running soon. Mainly Sjur’s task.

Decisions made in Tromsø can be found in [this meeting memo.|/admin/physical_meetings/tromso-2006-08-propnoun.html]

TODO:

fix bugs in lexc2xml; add comments to the log element (Saara)
finish first version of the editing (Sjur)
test editing of the xml files. If ok, then: (Sjur, Thomas, Trond)
make terms-smX.xml <=== automatically from propernoun-sme-lex.xml (add nob as well) (the morphological section should be kept intact, in e.g. propernoun-sme-morph.txt) (Sjur, Saara)
convert propernoun-($lang)-lex.txt to a derived file from common xml files (Sjur, Tomi, Saara)
implement data synchronisation between risten.no and the cvs repo, and possibly other servers (ie the G5 as an alternative server to the public risten.no - it might be faster and better suited than the official one; also local installations could be treated the same way)
start to use the xml file as source file
clean terms-sme.xml such that all names have the correct tag for their use (e.g. @type=secondary) (Thomas, Maaren, linguists)
merge placenames which are errouneously in different entries: e.g. Helsinki, Helsingfors, Helsset (linguists)
publish the name lexicon on risten.no (Sjur)
add missing parallel names for placenames (linguists)
add informative links between first names like Niillas and Nils (linguists)

9. Spellers

OOo spellers

Tomi is working on the lexicon conversion to the Hunspell format. It is moving forward.

TODO:

add Hunspell data conversion (Tomi)

Testing

Spelling Error Markup

TODO:

Set up ways of adding meta-information (source info, used in testing or not, added to lexicon or not) (Saara)
move Steinar’s error markup in the xml files to (a copy of) the original (Børre)

Automated testing

TODO:

document the AppleScript testing tool (Sjur)
improve speller test bench (Sjur)
- integrate the ccat speller testing options in the Makefile (Sjur)
add extraction of all known spelling errors in the corpus (not the prooftest corpus), and check that they are properly corrected (Børre, Sjur)
how to do a regression self-test:
- extract all the base forms in the lexicon, and run them through the speller
- extract all SUB-marked entries, and run them through the speller
  - integrate these in the make file (Sjur)

Lexicon conversion to the PLX format

It seems to work quite ok now. We might still consider to ask Xerox for a license.

New public beta

Delayed till the majority of the present bugs are fixed.

10. Other

Corpus contracts

TODO:

publish corpus contracts and project infra as open-source on NoDaLi-sta (Sjur)

Bug fixing

When fixing bugs, record the version number containing the fix in the Bugzilla bug report, such that for each bug, we know exactly when it should have been fixed, in what file(s) and what version.

55 open Divvun/Disamb bugs (21 of these 56 are speller bugs, 35 are general bugs), and 23 risten.no bugs

New team member

Ilona is joining the Divvun project from today - welcome! Hi! :)

Board meeting

Sjur went to Oslo on Thursday Aug. 16 to meet the board. Some highlights:

Additions to our project accepted by the board:

smj hyphenation
InDesign speller

The planned release party will be on December 11, in Oslo. All project members are invited.

Project meeting

We’ll meet in September, 24-28, to work on the hardest remaining issues. Default location is Kautokeino, but we’ll also consider Tromsø until the next project meeting.

11. Next meeting, closing

The next meeting is 27.8.2007, 09:30 Norwegian time.

The meeting was closed at 10:57.

Appendix - task lists for the next week

Boerre

run all known spelling errors in the prooftest corpus through the speller
add extraction of all known spelling errors in the regular corpus (not the prooftest corpus), and check that they are properly corrected
move Steinar’s error markup in the xml files to (a copy of) the original
fix bugs!

Ilona

lexicalise missing words
add sme names from FIN

Maaren

lexicalise actio compounds

Per-Eric

expand the smj typos list
add missing smj words
lexicalise words from the Olavi missing list
add compounding tags to adjectives

Saara

add new XSL/XML headers for proofing test docs
fix bugs!

Sjur

improve speller test bench:
- run all known spelling errors in the corpus through the speller
- document the AppleScript testing tool
- integrate regression self tests with the make file
- integrate the ccat speller testing options in the make file
publish corpus contracts and project infra as open-source on NoDaLi-sta
fix stuorra-oslolaš lower case o
ä/æ in smj speller
restart work on the XML name editor/risten.no integration
plan the rest of the project period
resend smj speller-correct document to Kåre Tjikkom
fix bugs!

Thomas

work with compounding
fix stuorra-oslolaš lower case o
ä/æ in smj speller
fix bugs!

Tomi

make PLX conversion test sample; add conversion testing to the make file
add Hunspell data generation/conversion
fix bugs!

Trond

Work on the web corpus issues
update the smj proper noun lexicon, and refine the morphological analysis, cf. the propernoun-smj-lex.txt
fix stuorra-oslolaš lower case o
add sma texts to the corpus repository
ä/æ in smj speller
fix bugs!.