Subject: Success story: XEmacs and UTF-8

From: Gilbert Baumann <[email-masked]>
Date: Tue, 20 Jul 1999 15:10:53 +0200

Hi,

this morning I managed to teach my version of XEmacs (20.4 + mule) the
UTF-8 encoding. Unfortunatly you need its sources to be able to patch
it.

First you need these files provided by MORIOKA Tomohiko:

http://turnbull.sk.tsukuba.ac.jp/Tools/XEmacs/xemacs-21.0-b55-emc-b55-ucs.diff
http://turnbull.sk.tsukuba.ac.jp/Tools/XEmacs/xemacs-ucs-conv-0.1.tar.gz

The .diff is a diff against the C sources. The tar ball is elisp code, 
which provides lots of code tables to map to and from Unicode. As the
name of the diff file suggests it is against XEmacs-21; I needed to
help `patch' a bit. The most notable difference to my XEmacs-20.4
sources is that file-coding.[ch] was called mule-coding.[ch].

For those unfamilar with the XEmacs-MULE stuff (as I am) a quick
guide:

What we call an encoding is calls MULE a 'coding-system'. The most
important commands are:

M-x set-file-coding-system
M-x set-buffer-process-coding-system	[comint buffers]

and the variable `file-coding-system-alist', which guides `find-file'
to guess the encoding used. After stuff was running, the very first
thing I did was this [1]:

  (setq file-coding-system-alist
	(append file-coding-system-alist
		(list '("" . gb-find-file-coding-system))))
  
  (defun gb-file-substring (filename start end)
    (let ((buffer (generate-new-buffer " *GB-TEMP*"))
	  (tmp nil)
	  (res nil))
      (save-excursion
	(set-buffer buffer)
	(let ((coding-system-for-read 'no-conversion))
	  (setq tmp (ignore-errors
		      (insert-file-contents filename nil start end nil)))
	  (cond (tmp
		 (setq res (buffer-substring 1 (+ 1 (cadr tmp)))))
		(t
		 (setq res nil)))))
      (kill-buffer buffer)
      res))

  (defun gb-fetch-encoding-header-field (filename)
    (let ((head (gb-file-substring filename 0 600)))
      (and head
	   (and (string-match "-\\*- \\([^*]*\\) -\\*-" head)
		(let* ((q (match-string 1 head))
		       (enc (and (string-match " Encoding: \\([^;]+\\);" q)
				 (match-string 1 q))))
		  (and enc
		       (setf enc (downcase enc))
		       (setf enc (intern-soft enc))
		       (find-coding-system enc)
		       enc))))))

  (defun gb-find-file-coding-system (mode filename)
    (cond ((eq mode 'insert-file-contents)
	   (gb-fetch-encoding-header-field filename))
	  (t
	   nil)))

This code looks into the special mode line introduced by -*- somewhere
in the first 600 bytes of the file about to opened; if now there is a
field "Encoding: xyz;" and the xyz encoding^[b^[dcoding system
exists, choose that. So now you could do e.g.

;;; -*- Mode: Lisp; Syntax: Common-Lisp; Package: CLEX; Encoding: utf-8; -*-

and XEmacs goes into utf-8 mode here.

Atfer everything was running I defined \u+03BB (greek lambda) as a
macro like:

(defmacro \u+03BB (x) `(lambda .,x))

Gilbert.

[1] Besides turning off the menubar, the scrollbars, the 3D modeline
    and all that glitter, which irritates me. And switching the
    background color to 'white'.