Evaluation of NLP systems: References

Linguateca

This list of references (URL: /aval_conjunta/bibEval.html) was prepared for my Tutorial on evaluation of NLP systems at SBIA/IBERAMIA 2000, São Paulo, Brasil, 19 November 2000. Its address (no longer valid) was http://www.portugues.mct.pt/Diana/bibTutEval.html.

The first version was published on the 15th November 2000. During spring and summer 2002 it was common work with Alexsandro Santos Soares, after the merging of his reference list on evaluation contests, and was moved to the present location, devoted to the preparation of evaluation campaigns for Portuguese. From September 2002 on, it was maintained only by Diana Santos again.


Books, proceedings, journals

Computer Speech and Language 12 (1998), Academic Press. Special issue on "Evaluation in Language and Speech Technology".

EAGLES, Evaluation of Natural Language Processing Systems, FINAL REPORT, EAGLES DOCUMENT EAG-EWG-PR.2 Version of September 1995, http://issco-www.unige.ch/ewg95/ewg95.html.

Proceedings of The First International Conference on Language Resources and Evaluation (Granada, 28-30 May 1998), 2 Volumes, ELRA.

Proc. Second International Conference on Language Resources and Evaluation, LREC'2000 (Athens, 31 May - 2 June 2000), 3 Volumes, ELRA.

Computers and the Humanities 34 (2000), Kluwer Academic Pusblishers. Special issue on "SENSEVAL: evaluating Word Sense Disambiguation Programs", guest editors: Adam Kilgarriff & Martha Palmer.

Sparck-Jones, Karen & Gallier, J. R. Evaluating natural language processing systems: An analysis and review, Berlin-Heidelberg-New York: Springer, 1995.

Cohen, Paul. Empirical Methods for Artificial Intelligence, Cambridge, Mass./London, England: MIT Press, 1995.

Papers on evaluation of NLP systems

Bailly, Gérard, Eduardo R. Banga, Alex Monaghan & Erhard Rank. "The Cost258 Signal Generation Test Array", Proc. Second International Conference on Language Resources and Evaluation, LREC'2000 (Athens, 31 May - 2 June 2000), Vol 2, pp. 651-4.

Bertier, Marc & Lallich-Boidin, Geneviève. "A paradox raised by the evaluation of taggers", in Antonio Rubio, Natividad Gallardo, Rosa Castro and Antonio Tejada (eds.), Proceedings of The First International Conference on Language Resources and Evaluation (Granada, 28-30 May 1998), Vol. 1, pp. 443-6.

Black, E., S. Abney, D. Flickinger, C. Gdaniek, R. Grishman, P. Harrison, D. Hindle, R. Ingria, F. Jelinek, J. Klavans, M. Liberman, M. Marcus, S. Roukos, B. Santorini & T. Strzalkowski. "A procedure for quantitatively comparing the syntactic coverage of English grammars", Proceedings of the February 1991 DARPA Speech and Natural Language Workshop (Pacific Grove, CA, February 1991), pp. 306-311.

Braschler, Martin and Peters, Carol. "The CLEF Campaign". Proceedings of the Second NTCIR Workshop on Research in Chainese & Japanese Text Retrieval and Text Summarization. National Institute of Informatics. Tokyo, Japan, 2001. http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings2/Martin.pdf

Budanitsky, Lexander & Graeme Hirst. ``Semantic distance in WordNet: An experimental, application-oriented evaluation of five measures.'' In Workshop on WordNet and Other Lexical Resources, Second meeting of the North American Chapter of the Association for Computational Linguistics (Pittsburgh, June 2001), available from http://www.cs.toronto.edu/compling/Publications/Abstracts/Papers/Budanitsky+Hirst-2001-abs.html .

Carletta, Jean. "Assessing Agreement on Classification Tasks: The Kappa Statistic", Computational Linguistics 22 (1996), pp. 249-54.

Carroll, John, Ted Briscoe & Antonio Sanfilippo. "Parser evaluation: a Survey and a New Proposal", in Antonio Rubio, Natividad Gallardo, Rosa Castro and Antonio Tejada (eds.), Proceedings of The First International Conference on Language Resources and Evaluation (Granada, 28-30 May 1998), Vol. 1, pp. 447-54.

Chinchor, Nancy A. "Overwiew of MUC-7/MET-2". Proceedings of the Seventh Message Understanding Conference http://www.itl.nist.gov/iaui/894.02/related_projects/muc/proceedings/muc_7_proceedings/overview.html

Fukumoto, Jun'ichi and Kato, Tsuneaki. "An Overview of Question and Answering Challenge (QAC) of the Next NTCIR Workshop". Proceedings of the Second NTCIR Workshop on Research in Chainese & Japanese Text Retrieval and Text Summarization. National Institute of Informatics. Tokyo, Japan, 2001. http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings2/fukumoto.pdf

Gaizauskas, Robert. "Evaluation in language and speech technology", Computer Speech and Language, 12 (1998), pp. 249-62.

Gaizauskas, R., M. Hepple & C. Huyck. "A Scheme for Comparative Evaluation of Diverse Parsing Systems", Proceedings of the 1st International Conference on Language Resources and Evaluation (LREC'98), Granada, 1998, pp. 143-149, http://www.dcs.shef.ac.uk/~robertg/publications/papers_by_topic.html#Evaluation in NLP

Gaizauskas, R., M. Hepple & C. Huyck. "Modifying Existing Annotated Corpora for General Comparative Evaluation of Parsing", Workshop on Evaluation of Parsing Systems, at the 1st International Conference on Language Resources and Evaluation (LREC'98) (Granada, 1998), http://www.dcs.shef.ac.uk/~robertg/publications/papers_by_topic.html#Evaluation in NLP

Grefenstette, Gregory. "Evaluation Techniques for Automatic Semantic Extraction: Comparing Syntactic and Window Bass Approaches", Workshop on Acquisition of Lexical Knowledge from Text (Columbus, OH, 21 June 1993), SIGLEX/ACL.

Guessoum, Ahmed & Rached Zantout. "A Methodology for a Semi-Automatic Evaluation of the Lexicons of Machine Translation Systems". Machine Translation , 16 (2) (2001), pp.127-49.

H.M. Harmain & R. Gaizauskas. "CM-Builder: An Automated NL-Based CASE Tool", The Fifteenth IEEE International Conference on Automated Software Engineering (ASE'00), September 11 - 15, 2000, Grenoble, France, pp. 45-54.

Harman, Donna. "Overview of the First Text REtrieval Confererence (TREC-1)". In Proceedings of the First Text REtrieval Confererence (TREC-1). Gaithersburg, Maryland, November 4-6, 1992. http://trec.nist.gov/pubs/trec1/papers/01.txt

Harman, Donna. "Overview of the Second Text REtrieval Confererence (TREC-2)". In Proceedings of the Second Text REtrieval Confererence (TREC-2). Gaithersburg, Maryland, August 31-September 2, 1993. http://trec.nist.gov/pubs/trec2/papers/ps/overview.ps.gz

Harman, Donna. "Overview of the Third Text REtrieval Confererence (TREC-3)". In Proceedings of the Third Text REtrieval Confererence (TREC-3). Gaithersburg, Maryland, November 2-4, 1994. http://trec.nist.gov/pubs/trec3/overview.ps.gz

Harman, Donna. "Overview of the Fourth Text REtrieval Confererence (TREC-4)". In Proceedings of the Fourth Text REtrieval Confererence (TREC-4). Gaithersburg, Maryland, November 1-3, 1995. http://trec.nist.gov/pubs/trec4/overview.ps.gz

Hausser, Roland. "The coordinator's final report on the first Morpholympics" LDV-Forum 11(1), 1994, pp. 54-64.

Heidorn, George. "Experience with an Easily Computed Metric for Ranking Alternative Parses", in Karen Jensen, George E. Heidorn & Stephen D. Richardson (eds), Natural Language Processing: The PLNLP Approach, Kluwer Academic Press, 1993, pp. 47-52.

Hindle, Donald & Mats Rooth. "Structural Ambiguity and Lexical Relations", Computational Linguistics 19 (1993), pp. 103-120.

Hirschman, Lynette. "Language Understanding Evaluations: Lessons Learned from MUC and ATIS", Proceedings of The First International Conference on Language Resources and Evaluation (Granada, 28-30 May 1998), Vol. 1, pp.117-122.

Hirschman, Lynette. "The evolution of Evaluation: Lessons from the Message Understanding Conferences", Computer Speech and Language 12 (1998), pp. 281-305.

Hirschman, Lynette, Patricia Robinson, John Burger & Marc Vilain. "Automating Coreference: The Role of Annotated Training Data", AAAI 1998 Spring Symposium on Applying Machine Learning to Discourse Processing, Stanford, pp. 1419-1422.

Hunt, Melvyn J. "Practical Large-Vocabulary Speech Recognition in a Multilingual Environment", Speech Communication 23 (1997), pp. 297-305.

Itahashi, Shuichi. "Guidelines for Japanese Speech Synthesizer Evaluation", Proc. Second International Conference on Language Resources and Evaluation, LREC'2000 (Athens, 31 May - 2 June 2000), Vol 2, pp. 655-60.

Jekat, Susanne J. & Lorenzo Tessiore. "End-to-End Evaluation of Machine Interpretation Systems: A Graphical Evaluation Tool", Working Papers in Multilingualism 4 (2000), Sonderforschungsbereich 538, Universität Hamburg.

Kando, Noriko. "CLIR System Evaluation at NTCIR Workshops". Proceedings of the Cross-Language System Evaluation 3 September, 2001. Darmstadt, Germany. http://www.ercim.org/publication/ws-proceedings/CLEF2/kando.pdf

King, Margaret. "Evaluating natural language processing systems", Communications of the ACM 39 (1996), pp. 73-79.

Lehnert, Wendy & Beth Sundheim. "A Performance Evaluation of Text-Analysis Technologies", AI Magazine 12 (1991), pp. 81-94.

Lewis, David D. "Evaluating text categorization", Proceedings of Speech and Natural Language Workshop, Morgan Kaufmann: San Mateo, CA, February 1991, pp.312-318.

Lin, Dekang. "A dependency-based method for evaluation broad-coverage parsers", Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence IJCAI'95, San Mateo, Calif: Morgan Kaufmann Publishers, pp.1420-5.

Macklovitch, Elliott. "Where the Tagger Falters", Proceedings of the 4th International Conference on Theoretical and Methodological Issues in Machine Translation (Montréal, June 25-27, 1992), pp. 113-26.

Mani, Inderjeet et alli. "TIPSTER SUMMAC Text Summarization Evaluation: Final Report". MITRE Technical Report. October, 1998. http://www-nlpir.nist.gov/related_projects/tipster_summac/summac-final-report-part2.ps

Mariani, J. "Some evaluation-based language engineering actions for French", Computer Speech and Language, 12 (4) (1998), pp.307-16.

Medeiros, José Carlos. "Avaliação de Correctores Ortográficos" [Evaluation of Spellcheckers], Actas do XI Encontro da Associação Portuguesa de Linguística (Lisboa, 2-4 de Outubro de 1995), Lisboa: Colibri, 1996, pp. 73-91.

Mellish, Chris & Robert Dale. "Evaluation in the context of natural language generation", Computer Speech and Language 12 (1998), pp. 349-73.

Mitkov, Ruslan. "Towards More Comprehensive Evaluation in Anaphora Resolution", Proc. Second International Conference on Language Resources and Evaluation, LREC'2000 (Athens, 31 May - 2 June 2000), Vol 3, pp.1309-14.

Mochizuki, Hajime & Manabu Okumura. "A Comparison of Summarization Methods Based on Task-based Evaluation", Proc. Second International Conference on Language Resources and Evaluation, LREC'2000 (Athens, 31 May - 2 June 2000), Vol 2, pp. 633-39.

Oepen, S. & Dan P. Flickinger. "Towards systematic grammar profiling. Test suite technology 10 years after", Computer Speech and Language 12 (1998), pp. 411-35.

Oepen, Stephan, Klaus Netter & Judith Klein. "TSNLP - Test Suites for Natural Language Processing", John Nerbonne (ed.), Linguistic Databases, CSLI Publications, 1988, pp. 13-36.

Pallett, David S. "The NIST Role in Automatic Speech Recognition Benchmark Tests", in Antonio Rubio, Natividad Gallardo, Rosa Castro and Antonio Tejada (eds.), Proceedings of The First International Conference on Language Resources and Evaluation (Granada, 28-30 May 1998), Vol. 1, pp. 327-30.

Patrick Paroubek, Marc Blasband (eds.), "ELSE LE4-8340, Evaluation in Language and Speech Engineering: Executive Summary of a Blueprint for a General Infrastructure for Natural Language Processing Systems Evaluation Using Semi-Automatic Quantitative Black Box Approach in a Multilingual Environment", http://www.limsi.fr/TLP/ELSE/FullXreportXver302.htm.

Ravin, Yael. "Disambiguating and Interpreting Verb Definitions", in Karen Jensen, George E. Heidorn & Stephen D. Richardson (eds), Natural Language Processing: The PLNLP Approach, Kluwer Academic Press, 1993, pp. 176-89.

Resnik, Philip. "Evaluating Multilingual Gisting of Web Pages", in Natural Language Processing for the World Wide Web, Papers from the 1997 AAAI Spring Symposium (Stanford, March 24-26, 1997), Menlo Park, California: AAAI Press, pp. 129-135.

Richardson, Stephen & Lisa Braden-Harder. "The Experience of Developing a Large-Scale Natural Language Processing System: Critique", in Karen Jensen, George E. Heidorn & Stephen D. Richardson (eds), Natural Language Processing: The PLNLP Approach, Kluwer Academic Press, 1993, pp. 77-89.

Santos, Diana & Signe Oksefjell. "An evaluation of the Translation Corpus Aligner, with special reference to the language pair English-Portuguese", in Torbjørn Nordgård (ed.), NODALIDA'99, Proceedings from the 12th "Nordisk datalingvistikkdager". Trondheim, 9-10 December 1999, Trondheim, Department of Linguistics, NTNU, 2000, pp. 191-205.

Setzer, Andrea & Robert Gaizauskas. "A Pilot Study on Annotating Temporal Relations in Text", Proceedings of the Worskhop for Temporal and Spatial Information Processing (Toulouse, July 7th 2001), EACL-ACL 2001, ACL: Toulouse, pp.73-80.

Simard, Michel, George Foster, Marie-Louise Hannan, Elliott Macklovitch & Pierre Plamondon. "Bilingual text alignment: where do we draw the line?", in Simon Philip Botley, Anthony Mark McEnery & Andrew Wilson (eds), Multilingual Corpora in Teaching and Research, Amsterdam/Atlanta, GA: Rodopi, 2000, pp. 38-64.

Thompson, Henry S. & Chris Brew. "Automatic Evaluation of Computer Generated Text: Final Report on the TextEval Project", July 2, 1996, http://www.cogsci.ed.ac.uk/~chrisbr/papers/mt-eval-final.ps.

Tinsley, Howard E. A. & David J. Weiss. "Interrater Reliability and Agreement." In Howard E. A. Tinsley and Steven D. Brown (eds.), Handbook of Applied Multivariate Statistics and Mathematical Modeling, San Diego, CA: Academic Press, 2000, pp. 95-124.

Underwood, Nancy, Patrizia Paggio & Gurli Rohde (1995). "A methodology for evaluating Spelling Checker functionality: Developing test suites for Danish", Short papers presented at the Tenth Scandinavian Conference on Computational Linguistics (Helsinki, 29-30th May 1995), compiled by Kimmo Koskenniemi, pp. 76-85.

Véronis, Jean & Philippe Langlais. "Evaluation of paralell text alignment systems: the ARCADE project" in Jean Véronis (ed.), Parallel Text Processing, Dordrecht: Kluwer Academic Publishers, pp. 369-88.

Voorhees, Ellen M. and Harman, Donna. "Overview of the Fifth Text REtrieval Confererence (TREC-5)". In Proceedings of the Fifth Text REtrieval Confererence (TREC-5). Gaithersburg, Maryland, November 20-22, 1996. http://trec.nist.gov/pubs/trec5/papers/overview.ps.gz

Voorhees, Ellen M. and Harman, Donna. "Overview of the Sixth Text REtrieval Confererence (TREC-6)". In Proceedings of the Sixth Text REtrieval Confererence (TREC-6). Gaithersburg, Maryland, November 19-21, 1997. http://trec.nist.gov/pubs/trec6/papers/overview.ps.gz

Voorhees, Ellen M. and Harman, Donna. "Overview of the Seventh Text REtrieval Confererence (TREC-7)". In Proceedings of the Seventh Text REtrieval Confererence (TREC-7). Gaithersburg, Maryland, November 09-11, 1998. http://trec.nist.gov/pubs/trec7/papers/overview_7.pdf.gz

Voorhees, Ellen M. and Harman, Donna. "Overview of the Eighth Text REtrieval Confererence (TREC-8)". In Proceedings of the Eighth Text REtrieval Confererence (TREC-8). Gaithersburg, Maryland, November 17-19, 1999. http://trec.nist.gov/pubs/trec8/papers/overview_8.pdf

Voorhees, Ellen M. and Harman, Donna. "Overview of the Ninth Text REtrieval Confererence (TREC-9)". In Proceedings of the Ninth Text REtrieval Confererence (TREC-9). Gaithersburg, Maryland, November 13-16, 2000. http://trec.nist.gov/pubs/trec9/papers/overview_9.pdf

Voorhees, Ellen M. and Harman, Donna. "Overview of TREC 2001". In Proceedings of the Text REtrieval Confererence (TREC 2001). Gaithersburg, Maryland, November 13-16, 2001. http://trec.nist.gov/pubs/trec10/papers/overview_10.pdf

Voorhes, Ellen M. & Dawn M. Tice. "The TREC-8 Question Answering Track Evaluation", in E.M. Voorhes & D.K.Harman (eds.), NIST Special Publication 500-246: The Eighth Text REtrieval Conference (TREC 8), Department of Commerce, National Institute of Standards and Technology, 1999, pp. 83-106, http://trec.nist.gov/pubs/trec8/papers/qa8.pdf.

Voorhes, Ellen M. & Dawn M. Tice. "The TREC-8 Question Answering Track", Proc. Second International Conference on Language Resources and Evaluation, LREC 2000 (Athens, 31 May 2000), Vol III, pp. 1501-8.

Voorhes, Ellen M. "The Philosophy of Information Retrieval Evaluation", in Carol Peters (ed.), Results of the CLEF 2001 Cross-Language System Evaluation Campaign: Working Notes for the CLEF 2001 Workshop (3 September 2001, Darmstadt, Germany), 2002.

Walker, Marilyn A. & Johanna D. Moore. "Empirical studies in discourse", Computational Linguistics 23 (1997), pp. 1-12.

Walker, M.A., D.J. Litman, C.A.Kamm & A. Abella. "Evaluating spoken dialogue agents with PARADISE: Two case studies", Computer Speech and Language, 12 (1998), pp. 317-47.

Walker, Marilyn A., Hirschman, Lynette & Aberdeen, John. "Evaluation For Darpa Communicator Spoken Dialogue Systems". In Language Resources and Evaluation Conference, LREC , 2000. http://www.research.att.com/~walker/lrec-comm-final.ps

White, J.S. & T.A. O'Connell. "Evaluation in the ARPA machine translation program: 1993 evaluation", Proceedings of the Human Language Technology Workshop, San Francisco: Morgan Kaufmann, 1994, pp. 135-140.

Will, C.A. "Comparing human and machine performance for natural language information extraction: results for English microelectronics from the MUC-5 evaluation", Proceedings of the Fifth Message Understanding Conference (Baltimore, MD, August 1993), pp. 53-67.

Wilson, G., I. Mani, B. Sundheim & L. Ferro. "A multilingual approach to annotating and extracting temporal information". Proceedings of the Worskhop for Temporal and Spatial Information Processing (Toulouse, July 7th 2001), EACL-ACL 2001, ACL: Toulouse, pp.81-87.

Wittenburg, Kent and Jim Barnett. "Canonical representation in NLP system design: A critical evaluation", Proceedings of the Second Conference on Applied Natural Language Processing (ACL), 1988, pp. 253-9.

Woods, W.A., L.A. Bookman, A. Houston, R.J. Kuhn, P. Martin & G. Green. "Linguistic knowledge can improve information retrieval", Proceedings of the 6th Applied NLP Conference, 2000, pp. 262-7.

See also the list of references on the evaluation of processing of Portuguese on /aval_conjunta/bibevalport.html

References on evaluation of systems related to NLP

Such as information retrieval or search engines.

Belew, Richard K. & John Hatton. "RAVE Reviews: Acquiring relevance assessments from multiple users", in M. Hearst & H. Hirsh (eds.), Working notes of the AAAI Spring Symposium on Machine Learning in Information Access, March 1996, AAAI Press, 1996.

Joachims, Thorsten. "Evaluating Search Engines using Clickthrough Data", draft, February 19, 2002, available from http://www.cs.cornell.edu/People/tj/publications/joachims_02b.pdf.

References on user requirements, user involvement and usability

Bevan, N., & Macleod, M. "Usability measurement in context". Behaviour and Information Technology 13 (1994), pp. 132-145.

Bosert, J.L. Quality Functional Deployment: A Practitioner's Approach. NY: ASQC Quality Press, 1991.

Daly-Jones, O., Bevan, N., & Thomas, C. Handbook of User-centred Design, INUSE, EC Telematics Applications project IE 2016, 1997.

Dumas, J. S., & Redish, J. C. A practical guide to usability testing. New Jersey: Norwood, 1993.

Kirwan, B., & Ainsworth, L. K.. A Guide to task analysis. London: Taylor & Francis, 1992.

Landauer, T.K. The Trouble with Computers. Usefulness, Usability, and Productivity, Cambridge, MA: MIT Press, 1995.

Macaulay, L. Requirements engineering. London / New York: Springer, 1996.

Maguire, M. C. e. User-Centred Requirements Handbook (Report D5.3): HUSAT Research Institute, 1998.

Poulson, D., Ashby, M. and Richardson, S. J. (eds.). UserFit - A practical handbook on user-centred design for Assistive Technology, HUSAT Research Institute, 1996.

Preece, J. Human-computer interaction. Wokingham, England / Reading, Mass.: Addison-Wesley Pub. Co, 1995.

Pressman, R.S. Software Engineering: A Practitioner's Approach. McGraw-Hill, NY, 1992.

References on publication research and computer science guidelines

Armstrong, J. Scott. "Research on Scientific Journals: Implications for Editors and Authors", Journal of Forecasting 1 (1982), pp. 83-104.

Armstrong, J. Scott. "Peer Review for Journals: Evidence on Quality Control, Fairness, and Innovation", Science and Engineering Ethics 3 (1997), pp. 63-84.

Kitchenham, Barbara A., Shari Lawrence Pfleeger, Lesley M. Pickard, Peter W. Jones, Jarrett Rosenberg & David C. Hoaglin. "Guidelines for empirical research in software engineering", Version 6, 23 December 1999, accepted for publication.

Tichy, Walter F. "Should Computer Scientists Experiment More?", IEEE Computer (1998), pp. 32-40.

Other references for the tutorial

Atwell, Eric. The Language Machine: The Impact of Speech and Language Technologies on English Language Teaching, The British Council, July 1999.

Bacelar do Nascimento, Maria Fernanda, Amália Mendes & Diana Santos. "O corpus e a classificação sintáctica das palavras", Actas do 1.o Encontro de Processamento de Língua Portuguesa (Escrita e Falada) - EPLP'93 (Lisbon, 25-26 February 1993), pp.125-9.

Ferré, Frederick. Philosophy of technology. University of Georgia Press, 1995 [First edition, 1988, Englewood Cliffs, N.J.]

Johansson, Stig. Papers in Contrastive Linguistics and Language Testing Lund: CWK Gleerup, 1985.

Leech, Geoffrey. "Corpus Annotation Schemes", Literary and Linguistic Computing 8 (1993), pp. 275-81.

Ripley, B.D. Pattern Recognition and Neural Networks. Cambridge: Cambridge University Press, 1996.

Santos, Diana. "The importance of vagueness in translation: Examples from English to Portuguese", Romansk Forum 5 (1997), pp. 43-69. (Revised bilingual version published as "A relevância da vagueza para a tradução, ilustrada com exemplos de inglês para português" / "The relevance of vagueness for translation: Examples from English to Portuguese", TradTerm 5 (1998), Universidade de São Paulo, pp. 41-98.)

Santos, Diana. "Toward language-specific applications", Machine Translation 14 (2000).

Santos, Diana. "Processamento de linguagem natural: uma apresentação através das aplicações" [Natural language processing: a presentation through applications], in Elisabete Ranchhod (ed.), Tratamento das Línguas por Computador. Uma introdução à linguística computacional e suas aplicações Lisboa: Caminho, to appear (presumably in 2001).

Acknowledgements

I am grateful to Jan Håvard Skjetne for publications and foils on usability, to John Krogstie and Ketil Stolen for references on empirical research in computer science and to Odd-Wiking Ralph for the papers on publication research.

I am grateful to several attendees of the tutorial for calling my attention to shortcomings and/or further topics of interest. Instead of rewriting the tutorial, I created a page on feedback that tries to convey some of it.


Diana Santos
Last modified on November 10, 2003.
Comments, suggestions and additions to Diana.Santos@sintef.no