Meeting setup

• Date: 30.01.2006
• Time: 09.30 Norw. time
• Place: Wherever we are :-)
• Tools: iChat, SubEthaEdit

Agenda

1. Opening, agenda review
2. Reviewing the task list from two weeks ago
3. Documentation - divvun.no
4. Corpus gathering
5. Corpus infrastructure
6. Linguistics
7. name lexicon infrastructure
8. Other issues
9. Summary, task lists
10. Closing

1. Opening, agenda review, participants

Opened at 09:48.

Present: Børre, Sjur, Tomi, Trond

Absent: Maaren, Saara, Thomas

Main secretary: Børre

Agenda accepted as is.

2. Reviewing the task list from the last meeting

Børre

• send out contracts with accompanying letter
  • Not done
• Gather public texts, preferrably also parallel ones
  • Not done
• Contact Odin editor (Ove Sæth) to ask for source (and parallel) documents
  • Done
• Continue converting text from input format to our xml
  • Not done
• review code and documentation for corpus xsl files under version control
  • Not done
• fix bugs!
  • Not done

Maaren

• work with risten.no
• discuss with relevant people regarding seminar on proofing tools, normativity and SGL in February/March, including place.

Saara

• Convert the name lexicon from present format to xml for testing; final conversion pending the editing fascilities
  • done
• Refine language detection for Finnish
  • not done
• Finish the review of the hyphenation detection.
  • not done
• Review the handling of xsl-files in corpus infrastructure, including version control
  • in progress
• Do some testing for bug #211
  • not done
• optimize the preprocess script
  • not done
• Write/update user documentation for the corpus usage in preparation for the review in the project meetings next week.
  • done
• finalize an improved working version of the CGI and command line scripts for corpus additions
  • in progress
• xml2lexc update to handle complex names: construct entries like we have now from the different parts of a complex name entry
  • tomi will do the xml2lexc-script(?)
• update conversion from lexc to xml (proper names) with the latest refinements
  • not done
• fix bugs!

Sjur

• Follow up the lawyer treatment of the contracts
  • not done
• Project seminar
  • plan and make schedule with Trond
    • done
  • check which hotels SD has an agreement with
    • done
  • plan XQuery/XSLT training session
    • done
    • the whole seminar is done
• Lule Sámi twol problems, with Thomas and Trond
  • not done
• follow up on voice group-chat not working to Sámediggi
  • Test Marratech when the new Marratech server is in place
    • not done
• project planning with Trond, continued
  • also look at the development processes - specification and testing
    • not done
• Follow up on place names from Norge Digitalt
  • write an e-mail to or call Bjørn Olav Megard
    • not done
• Evaluate SFST as speller (and analyzer) lexicon
  • more thorough analysis than was possible in Guovdageaidnu
    • not done
• write a background document on the corpus contracts
  • not done
• continue proper name lexicon work and discussion
  • done a lot at the seminar
• public tender:
  • review offer from Finnut Consult AS
    • done, as well as asked for two other offers, then picked one (Finnut), and had a meeting with them, planning the whole PT process
• smj G3 issue with Thomas and Trond
  • not done
• sme G3 issue with Thomas and Trond
  • not done
• call EDD/Christian Emil Ore about national place name lexicon
  • not done
• risten.no/name lexicon development: fix bugs, continue development
  • done some, backed up and restarted the server; the backup isn’t completely valid, due to some strange bug/conflict between the version on the server and the version of the client. The XML files are ok though.
• fix bugs!

Thomas

On sick leave

Tomi

• Aspell: Continue working on the affix file & aspell
  • Contact aspell author (UTF-8 thing)
    • Not done
• corpus infrastructure:
  • dtd location (both public and internal)
    • Not done
  • cgi-admin script for adding xsl-files
    • Not done
• Document aspell and corpus infrastructure
  • Not done
• Specification for new catxml in C++
  • install and announce new ccat tool
    • Done
• new proper name lexicon
  • remove last part of complex names not used as simplex names
    • Not done
  • start looking at conversion of the name lexicon from present format to xml
    • Not done
  • discuss the new lexicon format in the newsgroup
    • Not done
  • Look into synchronisation of proper names with risten.no
    • Not done
  • meeting to arrive at final xml format
    • Participated the meeting
  • new version of xml2lexc (based on catxml, now ccat)
    • Not done
• hyphenation in corpus docs
  • Not done, but has Saara done it? Sjur: yes.
• comment review template made by Saara
  • Not done
• fix bugs!
  • Not done
• pick up backpacks after Xmas
  • Done

Trond

• Contact Odin editor (Ove Sæth) immediately to reopen contacts
  • Done.
• Project seminar
  • plan and make schedule with Sjur
  • Done.
  • check with Linda and Ilona whether we can start on Monday after lunch
  • Done.
• sign contract with Bibelselskapet for Norwegian parallel texts
  • Done.
• document corpus infrastructure, your part
• review corpus usage documentation (ccat)
  • Done.
• discuss the new lexicon format in the newsgroup
  • Done, but not extensively.
• smj G3 issue with Sjur and Thomas
  • Not done
• sme G3 issue with Sjur and Thomas
  • Not done
• fix bugs!
  • I think we made some progress on some of the number bugs, but we haven’t looked at the bugzilla base yet.

3. Documentation

Reviews

ccat review

Saara, Linda, Ilona, Trond

Conducted the review as part of the seminar, although Thomas and Maaren weren’t there. Outcome:

It works mostly as documented, a few glitches were found and corrected. The documentation is terse but concise (simple is beautiful, that is, in full accordance to KISS). ccat -h gives:

        -a              Print all text elements.
        -p              Print plain paragraphs. (default)
        -T              Print paragraphs with title type.
        -L              Print paragraphs with list type.
        -t              Print paragraphs with table type.
        -r <dir>        Recursively process directory dir and subdirs enountered.
        -h              Print this help message.

• (Trond:) I thought that -T printed titles, but it gave the same output as -a.
• (Tomi:) No, it gave you all regular paragraphs + titles, but no tables or lists. The latest version will give you only titles, lists, etc according to the specified option.
• (Sjur:) I would like a -v option, so we are able to identify which version to send a bug report for :-) The version number should be autoincremented on each CVS check-in/build, or manually set?

The [xml-based documentation was not completely up-to-date with the latest changes, fixed in the meeting.

Other documentation

• Børre: Informed about the forrest documentation: the documentation tree structure, how to add documents and get them into the menu.

4. Corpus gathering

Discussed briefly how the formalities should be implemented: signature, Websak, posting etc.

Collecting

See the previous meeting memo for what’s to be done.

TODO: Still a lot for Børre!

Odin

DONE: Trond, and then Børre to call Ove Sæth to re-establish contact.

Sæth to discuss with colleagues about how to implement the cooperation.

Bible texts

ccat -t zcorp/gt/sme/bible/ot/1Mos_09-01.doc.xml | less

This gives everything. What we want is to make a file-specific version of testament.xml, with these properties:

• The first column should be suppressed
• The second column should be marked number or something
• The third column should be marked header if the typographic code in the fist column indicates it is a header (1 or 3) and it should be marked text if the typographic code indicates it is text (is 5).

  sme$ls zcorp/orig/sme/bible/ot/
  1Mos_09-01.doc  Salmmat-_garvasat_0203.doc
  

  There are two new books in the paratext format waiting in the nob orig and nno orig directories.

TODO:

• write a paratext2xml converter
• convert smj NT to paratext
• ask to get fin and swe NT and OT in paratext format

We already have an embryonic converter: gt/script/testament.xsl Usage: xsltproc /path/to/testament.xsl bible-text.xml > converted-text.xml format.

5. Corpus infrastructure

Task list:

1. Include the xsl files under version control
   1. RCS version control is almost finished, but an issue with access control is still open. Discussed a bit in the meeting, but nothing conclusive. We’ll continue the discussion in the newsgroup.
2. Incorporate language detection as part of the corpus processing (Saara)
   1. Almost finished. Needs improved Finnish language model - presently it isn’t able to distinguish Finnish from Sámi (proving the family bonds:-)
3. we need to review whether only automatic hyphen detection is good enough, or whether manual post-processing in some form is needed. Delayed until we have some results to base the review on.
   1. Acceptable results: 90% of all real hyphens correctly tagged.
4. CGI-admin script to add xsl-file to a corpus file that doesn’t have one (Saara)

Things are moving forward, but still more work to do. The list is left as is.

6. Linguistics

Nothing today, our linguists are on sick leave or not participating. For the tasks and their status, see the previous meeting memo.

7. Name lexicon infrastructure

Complex names

Task list for this issue:

• make sure xml2lexc can handle complex names in ways compatible with our present tool chain (=reconstruct the lexc format we have now) (Tomi)
  • the resulting file format should be identical to our present prop-name file (=lexc), that can then be converted to our new xml format using the same script as for the regular names (Tomi or Saara, but only when the technical details are settled)
• Saara has added the analyzer as part of the preprocess, but it is slow, and needs to be optimized.

XML format

Tasks:

1. make a test lexicon for evaluating the format, set up the editing, and test it (Saara)
   1. Done
2. update conversion from lexc to xml to reflect new xml format (Saara)
   1. mostly done, some open questions left
3. testing of conversion
4. eXist as editor:
   1. develop the needed XQueries and interface
   2. synchronisation between risten.no and
   3. test whether eXist as editor is actually working well

8. Other

SGL Seminar

• SGL/normativity seminar
  • all members = potentially/likely all languages
    • not all languages, only North Sámi
  • date? As early as possible, end of February/beginning of March
  • place? Maaren will investigate

Technical issues

• The mac os / perl bug (at least Trond and Sjur has it, [Bugzilla #211|http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=211]):
  • utf8 “\xC4” does not map to Unicode at /Users/trond/gt/script/preprocess line
    1. This msg did not show up in 10.3 (perl 5.8.1), but does so in 10.4 (perl 5.8.6). It is probably a perl - OS mismatch. (Trond, Saara, Tomi)
    • 10.4 introduced support for locales in the shell (10.3 and earlier didn’t know about locales)
  • Test: the result of the last line should indicate whether this is a problem in cat or a Perl/OS mismatch.
  • Is this a problem with ccat?
    • It doesn’t seem so (3 min and still counting)
    • In the end, the bug turned up with ccat as well. I gave the command:

    • zcorp/gt/sme//xml	preprocess –abbr=bin/abbr.txt	lookup -flags
      mbTT -utf8 bin/sme.fst	lookup2cg	vislcg –grammar=src/sme-dis.rle
      –minimal	sort	less

    • and it (in the end) responded:

      1729 constraint rules
      utf8 "\xA1" does not map to Unicode at /home/trond/gt/script/preprocess line 109, <> chunk 12.
      

To ccat’s defence I must say that cat, in a similar situation, would have given far more error messages (hold on, testing still under way).

   preprocess file_name.txt - OK
   cat file_name.txt | preprocess - bug!!
   catxml file_name.xml | preprocess - ??
   ccat filename | preprocess - bug !!

This bug isn’t a high priority any more, because ccat behaves differently than cat, and because there is the possibility of avoiding cat when working locally.

BUG: close as Won’t fix.

Bug fixing

30 open bugs (and 25 risten.no bugs)

Norwegian ispell press release

The i18n section of Skolelinux plans a press release including a paragraph about our project. We will ask them to reformulate a couple of things, and remove the links they’ve included. Text submitted to them:

• Dette er den eneste kilden til retteprogram for norsk uavhengig av Microsoft, bl.a. det som brukes av Apple.

• Et separat prosjekt ved Sametinget er i gang for å utvikle samiske retteprogram, bl a for Linux. Se http://www.divvun.no for mer info.

9. Summary, task list

Børre

• send out contracts with accompanying letter
• Gather public texts, preferrably also parallel ones
• Continue converting text from input format to our xml
• review code and documentation for corpus xsl files under version control
• fix bugs!

Maaren

• work with risten.no
• discuss with relevant people regarding seminar on proofing tools, normativity and SGL in February/March, including place.

Saara

• continue discussion on the new lexicon format
• Refine language detection for Finnish
• Finnish the review of the hyphenation detection.
• Review the handling of xsl-files in corpus infrastructure, including version control
• Do some testing for bug #211
• Fix the preprocess script and optimize it by building an analyzator for the multi-part expressions.
• finalize an improved working version of the CGI and command line scripts for corpus additions
• update conversion from lexc to xml (proper names) with the latest refinements
• Try to add numeral treatment as part of the analyzator.
• Change character coding detection to paragraph-based.
• fix bugs!

Sjur

• Follow up the lawyer treatment of the contracts
• Lule Sámi twol problems, with Thomas and Trond
• follow up on voice group-chat not working to Sámediggi
  • Test Marratech when the new Marratech server is in place
• project planning with Trond, continued
  • also look at the development processes - specification and testing
• Follow up on place names from Norge Digitalt
  • write an e-mail to or call Bjørn Olav Megard
• Evaluate SFST as speller (and analyzer) lexicon
  • more thorough analysis than was possible in Guovdageaidnu
• write a background document on the corpus contracts
• continue proper name lexicon work and discussion
• public tender:
  • review offer from Finnut Consult AS
• smj G3 issue with Thomas and Trond
• sme G3 issue with Thomas and Trond
• call EDD/Christian Emil Ore about national place name lexicon
• risten.no/name lexicon development: fix bugs, continue development
• fix bugs!

Thomas

• work on North Sámi compounding and derivation
• review corpus usage documentation
• smj G3 issue with Sjur and Trond
• sme G3 issue with Sjur and Trond

Tomi

• Aspell: Continue working on the affix file & aspell
  • Contact aspell author (UTF-8 thing)
• corpus infrastructure:
  • dtd location (both public and internal)
  • cgi-admin script for adding xsl-files
• Document aspell and corpus infrastructure
• ccat: add a -v option - it should return the version of the tool
• new proper name lexicon
  • remove last part of complex names not used as simplex names
  • start looking at conversion of the name lexicon from present format to xml
  • discuss the new lexicon format in the newsgroup
  • Look into synchronisation of proper names with risten.no
  • new version of xml2lexc (based on catxml, now ccat)
    • xml2lexc update to handle complex names: construct entries like we have now from the different parts of a complex name entry
• comment review template made by Saara
• fix bugs!

Trond

• Work on corpus texts with Børre.
• 3-part compounds with Sjur and Thomas.
• smj G3 issue with Sjur and Thomas.
• sme G3 issue with Sjur and Thomas.
• fix bugs!

10. Next meeting, closing

06.02.2006 09:30

Closed at 12:03