Language Technology at UiT

The Divvun and Giellatekno teams build language technology aimed at minority and indigenous languages

View GiellaLT on GitHub divvungiellatekno/giellalt.uit.no

Meeting setup

Agenda

  1. Opening, agenda review
  2. Reviewing the task list from last week
  3. Documentation - divvun.no
  4. Corpus gathering
  5. Corpus infrastructure
  6. Infrastructure
  7. Linguistics
  8. name lexicon infrastructure
  9. Spellers
  10. Other issues
  11. Summary, task lists
  12. Closing

1. Opening, agenda review, participants

Opened at 10:00.

Present: Børre, Ilona, Per-Eric, Risten, Sjur, Thomas, Tomi, Trond

Absent: none

Agenda accepted as is.

2. Updated task status since last meeting

Børre

Ilona

Maaren

Per-Eric

Risten

Saara

Sjur

Thomas

Tomi

Trond

3. Documentation

Nothing new.

TODO:

4. Corpus infrastructure

Nothing.

5. Infrastructure

Børre has been working on moving giellatekno.uit.no and www.divvun.no to the G5. The plan is to finish the work tomorrow.

Sjur has installed eXist on the G5 (but no risten.no yet).

6. Linguistics

North Sámi

Proper nouns and hyphens: when the name derives, like in -laš, we should get a hyphenation point in front of the derivation, but we don’t. Example:

Oslo > os^lo*laš
* = missing hyphenation point

sme $ lookup -flags mbTT -utf8 bin/hyph-sme.fst
0%>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>100%
oslolaš
oslolaš Os^lolaš

sme$lookup -flags mbTT -utf8 bin/hyphrules-sme.fst
0%>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 100%
oslolaš
oslolaš os^lo^laš

One possible solutioln is to introduce a marker for derivational suffixes (and, by extension to other languages, also prefixes):

What about inflectional suffixes? > is used in sma and nob. This speaks against using it for derivation.

root*1*der*2*infl
    %>     %>
Der1 Der2 Der3
C+   VC*  CVC

l u  tags, stem
l l  diacr  here
=
g u  diacr  here
g l  wordform (w or w/o hyphen)

:0 - no hyphenation

If visisble hyphenation:

:- IFF _ C V :0 IFF _ C #

HYPH = read regex ( @\"$(TARGET)/bin/hyphrules-$(TARGET).fst\"  .o. \
					@\"$(TARGET)/bin/hyph-i$(TARGET).save\"     .o. \
					@\"$(TARGET)/bin/$(TARGET)-norm.fst\" ) ; \n

SPLRNONREC = read regex [ [@\"$(TARGET)/bin/derivation-filter.fst\"     ] .o. \
						  [@\"$(TARGET)/bin/spellernonrec-$(TARGET).save\"  ] .o. \
						  [@\"common/bin/downcase.fst\"                 ] .o. \
						  [@\"common/bin/remove-hyphen.fst\"            ] .o. \
						  [@\"$(TARGET)/bin/hyphrules-$(TARGET).fst\".i ]     \
						] ; \n

$(TARGET)/bin/spellernonrec-$(TARGET).fst: \
							common/bin/tag-not-save.fst \
							common/bin/downcase.fst \
							$(TARGET)/bin/spellernonrec-$(TARGET).save \
							$(TARGET)/bin/derivation-filter.fst \
							common/bin/remove-hyphen.fst \
							$(TARGET)/bin/hyphrules-$(TARGET).fst

os^lolaš
os^lolaš        Oslo+N+Prop+Plc+Der1+Der/laš+A+Sg+Nom
os^lolaš        Oslo+N+Prop+Plc+Der1+Der/laš+A+SgGenCmp+Cmpnd
os^lolaš        Oslo+N+Prop+Plc+Der1+Der/laš+A+SgNomCmp+Cmpnd
os^lolaš        Oslo+N+Prop+Plc+Der1+Der/laš+A+Attr
os^lolaš        Oslo+N+Prop+Plc+Der1+Der/laš+A+Attr+Cmpnd

os^lo^laš
os^lo^laš       os^lo^laš       +?

In smj, also inflections fall outside our hyphenation rules:

Basudissaj
Basudissaj      Ba^su^dissaj
Bájddárin
Bájddárin       Bájd^dárin
Bájddárin       Bájd^dárin
Heandarahkii
Heandarahkii    Hean^da^rahkii

TODO:

Lule Sámi

TODO:

7. Name lexicon infrastructure

This sub-project needs to get up and running soon. Mainly Sjur’s task.

Decisions made in Tromsø can be found in [this meeting memo.|/admin/physical_meetings/tromso-2006-08-propnoun.html]

TODO:

  1. set up Tomcat and risten.no on the G5 again (Sjur, Børre)
    1. install risten.no
      1. eXist is now installed
  2. fix bugs in lexc2xml; add comments to the log element (Saara)
  3. finish first version of the editing (Sjur)
  4. test editing of the xml files. If ok, then: (Sjur, Thomas, Trond)
  5. make terms-smX.xml <=== automatically from propernoun-sme-lex.xml (add nob as well) (the morphological section should be kept intact, in e.g. propernoun-sme-morph.txt) (Sjur, Saara)
  6. convert propernoun-($lang)-lex.txt to a derived file from common xml files (Sjur, Tomi, Saara)
  7. implement data synchronisation between risten.no and the cvs repo, and possibly other servers (ie the G5 as an alternative server to the public risten.no - it might be faster and better suited than the official one; also local installations could be treated the same way)
  8. start to use the xml file as source file
  9. clean terms-sme.xml such that all names have the correct tag for their use (e.g. @type=secondary) (Thomas, Maaren, linguists)
  10. merge placenames which are errouneously in different entries: e.g. Helsinki, Helsingfors, Helsset (linguists)
  11. publish the name lexicon on risten.no (Sjur)
  12. add missing parallel names for placenames (linguists)
  13. add informative links between first names like Niillas and Nils (linguists)

8. Proofing tools

Hunspell

TODO:

  1. Follow-up support for the sami languages in OpenOffice.org (Børre, Sjur)
  2. Hunspell lexicon conversion (Tomi, Børre)
  3. add hunspell testing to the make file (Sjur)
  4. debug and fix remaining conversion issues (Børre, Tomi)

Testing

Spelling Error Markup

TODO:

Automated testing

TODO:

Lexicon conversion to the PLX format

Open issues based on test results:

smj

svierigaládde
svierigaládde   svieriga#ládde+N+Sg+Nom

polardutkamin
polardutkamin   polar#dutkat+V+TV+Actio+Der3+Der/m+N+Ess
polardutkamin   polar#dutkat+V+TV+Actio+Der3+Der/m+N+Sg+Ine

svieriga-
svieriga-       svieriga+Cmpnd

polar-
polar-  polar+Cmpnd

482 - still problematic, 552 - still open, Svierigadárogielan - still rejected

sme

397 - not as good as it should be, 408 - fixed, 419, 425 - roman number, 431, 452 - miel is a prefix, 461 - ovda accepted, almost 50 % (17) gets correct suggestion, 489, 508 - fixed, 522, 524.

TODO:

InDesign tools

TODO:

Hyphenators

We should look into the possibility of generating pattern-based hyphenation for OOo. It shouldn’t be too hard, or require too much work, but needs investigation.

TODO:

Release version

The CD cover etc. will be worked on by John-Marcus Kuhmunen, and will follow the SD design rules. He is now waiting for the text to be put on the CD cover and other places.

There is a bug when installing from CD over an old installation.

TODO:

Actual release

December 11-13, one of these days.

9. Other

Corpus contracts

Delayed till after final release.

TODO:

Faroese

Speller for fao using our infrastructure and the knowledge we have.

TODO:

Bug fixing

When fixing bugs, record the version number containing the fix in the Bugzilla bug report, such that for each bug, we know exactly when it should have been fixed, in what file(s) and what version.

69 open Divvun/Disamb bugs (35 of these 56 are speller-related bugs, 34 are other bugs), and 23 risten.no bugs

SD yearly personell seminar

6.-7. December. Sjur will discuss it with Julia, but our view is that we don’t have time to go this year. The release are just a few days later.

Software updates

10. Next meeting, closing

The next meeting is 12.11.2007, 09:30 Norwegian time.

The meeting was closed at 11:37.

Appendix - task lists for the next week

Boerre

Ilona

Maaren

Per-Eric

Risten

Saara

Sjur

Thomas

Tomi

Trond