[English | Japanese]

Namazu User's Manual


You can get the latest news about Namazu at http://www.namazu.org/. Namazu is free software under the terms of the GNU General Public License version 2 with ABSOLUTELY NO WARRANTY.

Table of contents

Namazu components

Namazu is a full-text search engine. Namazu uses the index maker mknmz command and the text searcher namazu command.

For quickly searching through many documents, Namazu generates an index similar to that of a book's.

mknmz command compiles the index. The target directory for indexing is given as an argument for mknmz. For example, if the target directory is /home/foo/public_html, then type

% mknmz /home/foo/public_html

Now documents such as *.html and *.txt under /home/foo/public_html are indexed and NMZ.* files are created in the directory where you run mknmz. NMZ.* files are from Namazu's index.

The namazu command searches the index. For example:

% namazu bar /home/foo/Namazu/foobar

The above searches a keyword "bar" for the index under /home/foo/Namazu/bar.

mknmz command

mknmz's options

    mknmz 2.0.X, an indexer of Namazu.

    Usage: mknmz [options] <target>...

    Target files:
      -a, --all                target all files.
      -t, --media-type=MTYPE   set the media type for all target files to MTYPE.
      -h, --mailnews           same as --media-type='message/rfc822'
          --mhonarc            same as --media-type='text/html; x-type=mhonarc'
      -F, --target-list=FILE   load FILE which contains a list of target files.
          --allow=PATTERN      set PATTERN for file names which should be allowed.
          --deny=PATTERN       set PATTERN for file names which should be denied.
          --exclude=PATTERN    set PATTERN for pathnames which should be excluded.
      -e, --robots             exclude HTML files containing
                               <meta name="ROBOTS" content="NOINDEX">
      -M, --meta               handle HTML meta tags for field-specified search.
      -r, --replace=CODE       set CODE for replacing URI.
          --html-split         split an HTML file with <a name="..."> anchors.
          --mtime=NUM          limit by mtime just like find(1)'s -mtime option.
                               e.g., -50 for recent 50 days, +50 for older than 50.

    Morphological Analysis:
      -b, --use-mecab          use MeCab for analyzing Japanese.
      -c, --use-chasen         use ChaSen for analyzing Japanese.
      -k, --use-kakasi         use KAKASI for analyzing Japanese.
      -m, --use-chasen-noun    use ChaSen for extracting only nouns.
      -L, --indexing-lang=LANG index with language specific processing.

    Text Operations:
      -E, --no-edge-symbol     remove symbols on edge of word.
      -G, --no-okurigana       remove Okurigana in word.
      -H, --no-hiragana        ignore words consist of Hiragana only.
      -K, --no-symbol          remove symbols.
          --decode-base64      decode base64 bodies within multipart entities.

    Summarization:
      -U, --no-encode-uri      do not encode URI.
      -x, --no-heading-summary do not make summary with HTML's headings.

    Index Construction:
          --update=INDEX       set INDEX for updating.
      -z, --check-filesize     detect file size changed.
      -Y, --no-delete          do not detect removed documents.
      -Z, --no-update          do not detect update and deleted documents.

    Miscellaneous:
      -s, --checkpoint         turn on the checkpoint mechanism.
      -C, --show-config        show the current configuration.
      -f, --config=FILE        use FILE as a config file.
      -I, --include=FILE       include your customization FILE.
      -O, --output-dir=DIR     set DIR to output the index.
      -T, --template-dir=DIR   set DIR having NMZ.{head,foot,body}.*.
      -q, --quiet              suppress status messages during execution.
      -v, --version            show the version of namazu and exit.
      -V, --verbose            be verbose.
      -d, --debug              be debug mode.
          --help               show this help and exit.
          --norc               do not read the personal initialization files.
          --                   Terminate option list.

    Report bugs to <http://www.namazu.org/trac-namazu/trac.cgi>
    or <bug-namazu@namazu.org>.

mknmzrc settings

Various settings are possible in mknmzrc or .mknmzrc. mknmzrc normally reads configuration files in the order of

  1. $(sysconfdir)/$(PACKAGE)/mknmzrc
    Usually, /usr/local/etc/namazu/mknmzrc
  2. ~/.mknmzrc
  3. file which is specified by -f or --config=FILE --option.

If more than one configuration file is found, they all of the files are loaded.

Installation prepares a sample configuration file $(sysconfdir)/$(PACKAGE)/mknmzrc-sample. You can copy this to $(sysconfdir)/$(PACKAGE)/mknmzrc or to ~/.mknmzrc in your home directory.

The setting details are given as comments in mknmzrc-sample.

Document filters

mknmz automatically identifies target file types and performs the appropriate document filtering. For HTML documents, filtering includes the extraction of <title> or the deletion of HTML tags. The filtering is dealt with by document filters in $(datadir)/$(PACKAGE)/filter. The standard document filters are described below.

apachecache.pl
Handles an Apache's cache file.
Requirement: None
Note: --replace=apachecache::replacecode option replaces to original URI
bzip2.pl
Handles a bzip2-ed file.
Requirement: bzip2 command.
compress.pl
Handles a compress-ed file.
Requirement: compress command.
deb.pl
Handles a deb package.
Requirement: dpkg command.
dvi.pl
Handles a dvi file.
Requirement: dvi2tty
Suggested software: nkf (only for Japanese documents)
excel.pl
Handles a Microsoft Excel file.
Dependence: html.pl
Requirement: xlhtml, (wvSummary, a part of wvWare)
Dependence module: extutf8.pl
gnumeric.pl
Handles a Gnumeric file.
Requirement: gzip command or Compress::Zlib perl module.
gzip.pl
Handles a gzipped file.
Requirement: gzip command or Compress::Zlib perl module.
hdml.pl
Handles a HDML file.
Requirement: None
hnf.pl
Handles a file of Hyper NIKKI System Project.
Requirement: the hnf filter is special: it requires namazu_for_hns of Hyper NIKKI System Project.
Dependence: html.pl
html.pl
Handles a HTML file.
Requirement: None
koffice.pl
Handles a KOffice KWord, KSpread, KPresenter, Kivio file.
Dependence module: extutf8.pl, extzip.pl
macbinary.pl
Handles a MacBinary I,II,III file.
Avoid a problem with handle a MacBinary file.
Requirement: None
mailnews.pl
Handles a file of Mail/News and MHTML file.
Requirement: None
Note: To handle MHTML file and Attached base64 bodies, MIME::Base64 and MIME::QuotedPrint are required.(perl5.8 contains them.) --decode-base64 option is required when handling a MHTML file or base64-encoded bodies.
man.pl
Handles a man file.
Requirement: nroff, groff or jgroff
Note: To handle Japanese man, groff supporting -Tnippon is required.
mhonarc.pl
Handles a MHonArc file.
Requirement: None
Dependence: html.pl, mailnews.pl
mp3.pl
Handles an MP3 file's ID3 Tag
Requirement: MP3::Info perl module. (version 1.01 or later are suggested).
msword.pl
Handles a Microsoft Word file.
Dependence: html.pl
Requirement: wvWare
Dependence module: extutf8.pl
msofficexml.pl
Handles an Microsoft Office2007 OfficeOpenXML Word, Excel, Powerpoint file.
Dependence: ooo.pl
Dependence module: extutf8.pl, extzip.pl
ooo.pl
Handles an OpenOffice.org Writer, Calc, Impress, Draw file.
Dependence module: extzip.pl
pdf.pl
Handles a PDF file.
Requirement: pdftotext, a part of xpdf (version 0.91 or later are suggested).
pipermail.pl
Handles a Mailman/pipermail file.
Requirement: None
Dependence: html.pl, mailnews.pl
postscript.pl
Handles a PostScript file.
Requirement: ps2ascii
powerpoint.pl
Handles a Microsoft PowerPoint file.
Dependence: html.pl
Requirement: pptHtml, a part of xlHtml, (wvSummary, a part of wvWare)
Dependence module: extzip.pl
rfc.pl
Handles an RFC file.
Requirement: None
rpm.pl
Handles an RPM package.
Requirement: rpm
rtf.pl
Handles a RTF file.
Requirement: rtf2html
Dependence: html.pl
taro56.pl
Handles a file of Ichitaro, a Japanese word processor, versions 5 and 6.
Requirement: None
taro7_10.pl
Handles a file of Ichitaro, a Japanese word processor, versions 7 through 13.
Requirement: unicode.pl, OLE::Storage_Lite perl module, IO::stringy perl module.
tex.pl
Handles a TeX file.
Requirement: detex
visio.pl
Handles a Microsoft Visio (.vdx) file.
Dependence: ooo.pl
Dependence module: extutf8.pl
zip.pl
Handles a Zip archive files.
Requirement: unzip
Alternative: Compress::Zlib perl module, Archive::Zip perl module.

The following filters are for Windows only.

ichitaro456.pl
Handles a file of Ichitaro, a Japanese word processor, versions 4, 5 and 6.
Requirement: JSTXT
Note: JSTXT is a tool for MS-DOS.
oleexcel.pl
Handles a Microsoft Excel file.
Requirement: Microsoft Excel 97 SP1 or later, 2000, 2002(XP), 2003 or 2007

Handles a Microsoft Excel 2007 file.
Requirement(A): Microsoft Excel 2007
Requirement(B): Microsoft Excel 2002(XP), 2003 and "Microsoft Office Compatibility Pack SP2 for Word, Excel, and PowerPoint 2007 File Formats"
olemsword.pl
Handles a Microsoft Word file.
Requirement: Microsoft Word 97 SP1 or later, 98, 2000, 2002(XP), 2003 or 2007

Handles a Microsoft Word 2007 file.
Requirement(A): Microsoft Word 2007
Requirement(B): Microsoft Word 2000, 2002(XP), 2003 and "Microsoft Office Compatibility Pack SP2 for Word, Excel, and PowerPoint 2007 File Formats"
olepowerpoint.pl
Handles a Microsoft PowerPoint file.
Requirement: Microsoft PowerPoint 97 SP1 or later, 2000, 2002(XP), 2003 or 2007

Handles a Microsoft PowerPoint 2007 file.
Requirement(A): Microsoft PowerPoint 2007
Requirement(B): Microsoft PowerPoint 2002(XP), 2003 and "Microsoft Office Compatibility Pack SP2 for Word, Excel, and PowerPoint 2007 File Formats"
oletaro.pl
Handles a file of Ichitaro, a Japanese word processor, versions 4.
Requirement: Microsoft Word 97 SP1 or later, 98 or 2000
Requirement: and applicable document converter of Microsoft Office attachment.
Dependence: olemsword.pl


Handles a file of Ichitaro, a Japanese word processor, versions 5 through 6.
Requirement: Microsoft Word 97 SP1 or later, 98, 2000 or 2002(XP)
Requirement: and applicable document converter of Microsoft Office attachment.
Dependence: olemsword.pl


Handles a file of Ichitaro, a Japanese word processor, versions 7 through 13, 2004.
Requirement: Microsoft Word 97 SP1 or later, 98, 2000, 2002(XP), 2003 or 2007
Requirement: and applicable document converter of Microsoft Office attachment.
Dependence: olemsword.pl
olertf.pl
Handles a RTF file.
Requirement: Microsoft Word 97 SP1 or later, 98, 2000, 2002(XP), 2003 or 2007
Dependence: olemsword.pl
olevisio.pl
Handles a Microsoft Visio file.
Requirement: Microsoft Visio 2000, 2002, 2003 or 2007
Dependence: olemsword.pl
xdoc2txt.pl
Handles a file of Microsoft Word, Excel, Powerpoint, Ichitaro, etc.
Requirement: xdoc2txt.exe
Note: xdoc2txt.exe is a tool for MS-Win32.

NOTE: We believe that mknmz will work well on both the English version and the Japanese version of Microsoft Office, but that is not yet confirmed. We would be grateful if you would notify us how it works. Thanks in advance.

The extension module of the following Namazu attachment is used with various document filters.

extutf8 module
The UTF-8 code is processed.
Requirement: NKF module, or Perl 5.8 or later, or lv command, or unicode.pl
extzip module
The zip compression data is handled.
Requirement: IO::Compress module, Archive::Zip module (Option IO::String module), or unzip command

namazu command

namazu's options

    namazu 2.0.X, a search program of Namazu.

    Usage: namazu [options] <query> [index]...
        -n, --max=NUM        set the number of documents shown to NUM.
        -w, --whence=NUM     set the first number of documents shown to NUM.
        -l, --list           print the results by listing the format.
        -s, --short          print the results in a short format.
            --result=EXT     set NMZ.result.EXT for printing the results.
            --late           sort the documents in late order.
            --early          sort the documents in early order.
            --sort=METHOD    set a sort METHOD (score, date, field:name)
            --ascending      sort in ascending order (default: descending)
        -a, --all            print all results.
        -c, --count          print only the number of hits.
        -h, --html           print in HTML format.
        -r, --no-references  do not display the reference hit counts.
        -H, --page           print the links of further results.
                             (This is nearly meaningless)
        -F, --form           force to print the <form> ... </form> region.
        -R, --no-replace     do not replace the URI string.
        -U, --no-decode-uri  do not decode the URI when printing in a plain format.
        -o, --output=FILE    set the output file name to FILE.
        -f, --config=FILE    set the config file name to FILE.
        -C, --show-config    print the current configuration.
        -q, --quiet          do not display extra messages except search results.
        -d, --debug          be in debug mode.
        -v, --version        show the namazu version and exit.
            --help           show this help and exit.
            --norc           do not read the personal initialization files.
            --               Terminate option list.

    Report bugs to <http://www.namazu.org/trac-namazu/trac.cgi>
    or <bug-namazu@namazu.org>.

You can specify one or more target indices in a command-line argument [index dir].... If the target index is omitted, the Default index will be treated as the target index.

By prefixing + such as +foo or +bar, you can specify a target index as a relative path from the default index.

When executed from a command line, Namazu outputs query results in simple text format. The -h option is required in order to display query results in HTML format.

If you want to display query results from the 21st hit through the 40th, type -n 20 -w 20 on the command line. Note that -w is not 21 in this example.

namazurc settings

Various settings are possible in mknmzrc or .mknmzrc. Namazu normally reads configuration files in the following order:

  1. $(sysconfdir)/$(PACKAGE)/namazurc
    (Usually, /usr/local/etc/namazu/namazurc
  2. ~/.namazurc
  3. file which is specified by -f or --config=FILE --option.
    (In case of CGI, it is .namazurc in the directory namazu.cgi is stored)

If more than one configuration file is found, all of the files are loaded.

Installation prepares a sample configuration file $(sysconfdir)/$(PACKAGE)/namazurc-sample. You can copy this to $(sysconfdir)/$(PACKAGE)/namazurc or to ~/.namazurc in your home directory.

The setting details are given as comments in namazurc-sample.

Default Index

The default index is the index that is used when no other index is specifiedand it follows the rules described below.

In CGI(namazu.cgi), the index selection is given as a relative path from the default index.

namazu.cgi

namazu.cgi installation

namazu.cgi is the CGI script for Namazu. namazu.cgi is installed in $(libexecdir) directory (usually /usr/local/libexec). If you copy namazu.cgi into a CGI directory of your system, installation is done!

.namazurc settings

If you have a .namazurc file in the directory where namazu.cgi is stored, the .namazurc file will be treated as a CGI configuration file. For the display to be in Japanese, you need the following settings:

Lang ja

Template files

Template files explain the display styles of query results in HTML. The details are described below.

NMZ.head
Header of search results.
NMZ.foot
Footer of search results.
NMZ.body
Description of Namazu's query.
NMZ.tips
Tips on searching.
NMZ.result
Format of search results.

These files are available for either language. Files suffixed by .ja are for Japanese.

Form settings

Form is defined in NMZ.head. CGI variables are as follows:

query
specify a query expression.
max
specify the maximum number of query results to display at once.
result
specify the display style of query results.
sort
specify the sorting routine.
idxname
specify the name of the index to search.
subquery
specify the sub-query expression.
whence
specify where you wish to display query results.
reference
specify whether or not to display reference hit counts.
lang
specify language of search results.

Selecting an index

To select an index from the browser, NMZ.head needs the following.


      <strong>Target:</strong>
      <select name="idxname">
      <option selected value="foo">foo</option>
      <option value="bar">bar</option>
      <option value="baz">baz</option>
      </select>

In the above example, you can select a single index from foo, bar, or baz. When foo is selected, Namazu searches the foo index under the default index. In case the default index is /usr/local/var/namazu/index, there will be directories as follows.


       /
       + usr/
         + local/
           + var/
             + namazu/
               + index/
                 + foo/
                 + bar/
                 + baz/

Selecting multiple indices

For selecting multiple indices NMZ.head needs a checkbox.


      <strong>Target</strong>
      <ul>
      <li><input type="checkbox" name="idxname" value="foo">foo</li>
      <li><input type="checkbox" name="idxname" value="bar">bar</li>
      <li><input type="checkbox" name="idxname" value="baz">baz</li>
      </ul>

In the above example, you can select multiple indices from foo, bar, and/or baz. The template file specified in the namazurc Template directive is used in searching. If no file is given in the Template directive, the following rules apply.

Using an auxiliary query

You can set an auxiliary query apart from a user-inputing query. The following example describes a way to limit target pages by URI.


      <strong>Target</strong>
      <select name="subquery">
      <option value="">All</option>
      <option value="+uri:/^http://foo.example.jp/foo//">foo's pages</option>
      <option value="+uri:/^http://foo.example.jp/bar//">bar's pages</option>
      <option value="+uri:/^http://foo.example.jp/baz//">baz's pages</option>
      <option value="+uri:/^http://foo.example.jp/quux//">quux's pages</option>
      </select>
      <input type="hidden" name="reference" value="off">

Selecting the language of search results

To select the language of search results on your Web browser, you can set NMZ.head with CGI variable lang as follows:


      <strong>Language:</strong>
      <select name="lang">
      <option selected value="ja">Japanese</option>
      <option value="">English</option>
      </select>

NOTE: Lang directive in .namazurc has precedence over the CGI variable lang. Do not set the Lang directive in .namazurc if you want to use the CGI variable lang.

Included tools

bnamazu

bnamazu is a search tool that operates with Web browsers. The query results are passed to a Web browser (default: lynx ) and from there to the users. The command line options are as follows.

% bnamazu [-n] [-b browser] [namazu's options] <query> [index]...

The -b option specifies a Web browser. The -n option is valid only when using netscape and it opens a new netscape window to display the query results.

nmzgrep

nmzgrep is a search tool that operates with the egrep command. nmzgrep executes egrep for retrieved documents. By applying egrep, you can find the line numbers to places where the keyword is found. The command line options are as follows.

% nmzgrep [egrep's options] <pattern> [index]...

For example, in order to search the ~/Namazu/foobar index for foo and to apply egrep to the search documents, you can do as follows:

% nmzgrep foo ~/Namazu/foobar

gcnmz

When you update the indices after deleting or adding documents, garbage will be placed in index files. gcnmz is a tool for garbage collection. The command line options are as follows.

% gcnmz [options] <target>...

To run garbage collection for indices in ~/Namazu/foobar, type

% gcnmz ~/Namazu/foobar

mailutime

mailutime is a tool to set timestamps of Mail/News files to the Date: header. The command line options are as follows.

% mailutime <target>...

To change the time stamps of emails stored in ~/Mail/ml/foobar, type

% mailutime ~/Mail/ml/foobar/*

vfnmz

vfnmz is a tool to preview search results. The command line options are as follows.

% vfnmz <index> [NMZ.result.foobar]

To preview indices stored in ~/Mail/ml/foobar, type

% vfnmz ~/Namazu/foobar > foobar.html
% lynx foobar.html

rfnmz

rfnmz is a tool to reconstruct NMZ.field.*.i files. The usage is:

% rfnmz <index>

For example, to reconstruct NMZ.field.*.i files in the ~/Namazu/foobar index, you can do:

% rfnmz ~/Namazu/foobar

nmzegrep

nmzegrep is a search tool that operates with the egrep command and mknmz document filters. nmzegrep executes egrep for retrieved documents. By applying egrep, you can find the line numbers to places where the keyword is found. The command line options are as follows.

% nmzegrep [egrep's options] <pattern> [index]...

For example, in order to search the ~/Namazu/foobar index for foo and to apply egrep to the search documents, you can do as follows:

% nmzegrep foo ~/Namazu/foobar

NOTE: nmzegrep do egrep after passing the mknmz document filter for retrieved document. nmzegrep is slower than nmzgrep because it calls the document filter of mknmz. However, nmzegrep can do egrep for various file format.

nmzcat

nmzcat converts the binary files from various format to plain text.

% nmzcat <target>...

NOTE: nmzcat supports only the file of the type supported by mknmz.

Query

Single term query

This query specifies only one term for retrieving all of the documents which contain that term, for example:

namazu

AND query

This query specifies two or more terms for retrieving all of the documents which contain both terms. Insert the and operator between the terms, e.g.

Linux and Netscape

You can omit the and operator. Terms which are separated by one ore more spaces are assumed to be an AND query.

OR query

This query specifies two or more terms for retrieving all documents which contain any one term. Insert the or operator between the terms, e.g.

Linux or FreeBSD

NOT query

This query specifies two or more terms for retrieving all of the documents which contain a first term but do not contain the following terms. Insert the not operator between the terms, for example:

Linux not UNIX

Grouping

You can group queries by surrounding them by parentheses. The parentheses should be separated by one or more spaces. e.g.

( Linux or FreeBSD ) and Netscape not Windows

Phrase searching

You can search for a phrase that consists of two or more terms by surrounding them with double quotation marks or braces such as "..." and {...}. In Namazu, the precision of phrase searching is not 100 %, so wrong results may occasionally occur. Example:

{GNU Emacs}

Substring matching

The are three types of searching by substring matching.

Prefix matching
inter* (terms which begin with inter)
Inside matching
*text* (terms which contain text)
Suffix matching
*net (terms which terminated with net)

Regular expressions

You can use regular expressions for pattern matching. The regular expressions must be surrounded by slashes like /.../. Namazu uses Ruby's regular regular expressions engine. It generally offers a Perl compatible flavor. e.g.,

/pro(gram|blem)s?/

Field-specified searching

You can limit your search to specific fields such as Subject:, From:, Message-Id:. This feature is especially convenient for Mail/News documents, for example:

Notes


Namazu Homepage

$Id: manual.html.en,v 1.49 2009-09-20 18:11:18 opengl2772 Exp $
developers@namazu.org