Using Portindex to Generate INDEX-5

Now that we've seen how to keep the ports tree up-to-date using tools like Portsnap, and seen how to generate an INDEX-5 file with 'make index', I'd like to offer an alternative INDEX-5 generation mechanism. Matthew Seaman's Portindex is a Perl tool replacing a similar application of the same name. The old version was pulled from the ports tree when the developer started acting strangely.

Here's the problem Portindex solves. If you use CVSup to update your ports tree, you will not have an INDEX-5 file. The INDEX-5 file included in CVS was reportedly always out-of-date, so you had to issue a command like 'portsdb -U' or 'make index' to create the INDEX-5 file. This was very time-consuming.

Portindex is a much faster alternative, once you've set up the system. Portindex is available in the ports tree as sysutils/p5-FreeBSD-Portindex , so installation is easy (as long as your ports tree is updated!) My installation is based on Matthew's excellent documentation at his site and in the Portindex man pages.

Once installed, you need to ininitalize the cache Portindex uses:

janney:/root# cache-init
Processing make describe output for path "/usr/ports": .........[1000].........F
reeBSD::Portindex::Tree::_scan_makefiles(): Can't open Makefile in /usr/ports/de
vel/jude_community -- No such file or directory at /usr/local/bin/cache-init lin
e 100
[2000].........[3000].........[4000].........[5000].........[6000].........[7000
].........[8000].........[9000].......FreeBSD::Portindex::Tree::_scan_makefiles(
): Can't open Makefile in /usr/ports/textproc/csv2txt -- No such file or directo
ry at /usr/local/bin/cache-init line 100
..[10000].........[11000].........<11993>

This took a long time -- hours -- on my dual PIII system. Once done, I had these new files:

janney:/var/db/portindex$ ls -al
total 9966
drwxrwxr-x  2 root  operator      512 Nov 24 18:25 .
drwxr-xr-x  9 root  wheel         512 Nov 24 18:22 ..
-rw-r--r--  1 root  operator     8192 Nov 24 21:10 __db.001
-rw-r--r--  1 root  operator   270336 Nov 24 21:10 __db.002
-rw-r--r--  1 root  operator   344064 Nov 24 21:10 __db.003
-rw-r-----  1 root  operator  9617408 Nov 24 21:10 portindex-cache.db
-rw-r--r--  1 root  operator       25 Nov 24 18:25 portindex-timestamp

Next I had to update the ports tree and give Portindex the output in a form it could use:

janney:/root# script /tmp/cvsup.out cvsup -g -L2 -h cvsup2.freebsd.org
 /usr/local/etc/ports-supfile 
Script started, output file is /tmp/cvsup.out
Parsing supfile "/usr/local/etc/ports-supfile"
Connecting to cvsup2.freebsd.org
Connected to cvsup2.freebsd.org
Server software version: SNAP_16_1h
Negotiating file attribute support
Exchanging collection information
Establishing multiplexed-mode data connection
Running
Updating collection ports-all/cvs
 Delete ports/INDEX-5
 Edit ports/Tools/portbuild/scripts/buildscript
  Add delta 1.17 2004.11.25.00.02.38 kris
...edited...
 Checkout ports/x11-wm/ratpoison/files/ratpoison.desktop
 Edit ports/x11-wm/ratpoison/pkg-plist
  Add delta 1.6 2004.11.24.17.32.29 hq
Shutting down connection to server
Finished successfully

Script done, output file is /tmp/cvsup.out

Notice that at the end of this process, INDEX-5 is gone:

janney:/root# ls -al /usr/ports/IND*
-rw-r--r--  1 root  wheel  12250112 Nov 24 08:11 /usr/ports/INDEX.db

We saw earlier that CVS executed ' Delete ports/INDEX-5' when updating /usr/ports.

Now run 'cache-update' against the CVS output:

janney:/root#  cache-update -i /tmp/cvsup.out
cache-update:1: Updating cached data for /usr/ports/audio/daapd
cache-update:2: Updating cached data for /usr/ports/audio/libsidplay2
cache-update:3: Updating cached data for /usr/ports/chinese/MT
cache-update:4: Updating cached data for /usr/ports/chinese/zhcon
cache-update:5: Updating cached data for /usr/ports/comms/efax-gtk
cache-update:6: Updating cached data for /usr/ports/comms/libticables
cache-update:7: Updating cached data for /usr/ports/comms/minicom
cache-update:8: Updating cached data for /usr/ports/databases/myodbc
cache-update:9: Updating cached data for /usr/ports/databases/slony1
cache-update:10: Updating cached data for /usr/ports/deskutils/hot-babe
cache-update:11: Updating cached data for /usr/ports/devel/bison
cache-update:12: Updating cached data for /usr/ports/devel/gettext
...edited...
cache-update:78: Updating cached data for /usr/ports/x11-wm/ratpoison
cache-update:79: Updating cached data for /usr/ports/x11/xextensions

Finally run 'portindex' to create INDEX-5:

janney:/root# portindex -o /usr/ports/INDEX-5
Accumulating dependency information: .........[1000].........[2000].........[300
0].........[4000].........[5000].........[6000].........[7000].........[8000]...
......[9000].........[10000].........[11000].........[12000]<12000>
Writing INDEX file: .........[1000].........[2000].........[3000].........[4000]
.........[5000].........[6000].........[7000].........[8000].........[9000].....
....[10000].........[11000].........[12000]<12000>
janney:/root# ls -al /usr/ports/IND*
-rw-r--r--  1 root  wheel   5959708 Nov 24 21:47 /usr/ports/INDEX-5
-rw-r--r--  1 root  wheel  12250112 Nov 24 08:11 /usr/ports/INDEX.db

I'm not sure if I will continue to use this method. As I just described it, you must use CVSup. Matthew offers 'find-updated' as a way to scan the /usr/ports tree and discover differences. I believe this could be paired with Portsnap to let Portsnap update the tree and Portindex update INDEX-5. That will probably be another Blog entry...