Monday, November 29, 2004

FreeBSD Project Goals

After reading PHK's Why Bother? article, I wondered about the goals of the FreeBSD project. I found them in the handbook:

"1.3.2 FreeBSD Project Goals
Contributed by Jordan Hubbard.

The goals of the FreeBSD Project are to provide software that may be used for any purpose and without strings attached. Many of us have a significant investment in the code (and project) and would certainly not mind a little financial compensation now and then, but we are definitely not prepared to insist on it. We believe that our first and foremost 'mission' is to provide code to any and all comers, and for whatever purpose, so that the code gets the widest possible use and provides the widest possible benefit. This is, I believe, one of the most fundamental goals of Free Software and one that we enthusiastically support.

That code in our source tree which falls under the GNU General Public License (GPL) or Library General Public License (LGPL) comes with slightly more strings attached, though at least on the side of enforced access rather than the usual opposite. Due to the additional complexities that can evolve in the commercial use of GPL software we do, however, prefer software submitted under the more relaxed BSD copyright when it is a reasonable option to do so."

These appear to be unchanged since 1998 and probably earlier. If asked to summarize these two paragraphs, it seems the FreeBSD project's goal is to create free software.

Compare those goals to OpenBSD (edited slightly here):

"Provide the best development platform possible. Provide full source access to developers and users, including the ability to look at CVS tree changes directly.

Integrate good code from any source with acceptable copyright (ISC or Berkeley style preferred, GPL acceptable as a last recourse but not in the kernel, NDA never acceptable). We want to make available source code that anyone can use for ANY PURPOSE, with no restrictions. We strive to make our software robust and secure, and encourage companies to use whichever pieces they want to...

Pay attention to security problems and fix them before anyone else does. (Try to be the #1 most secure operating system).

Greater integration of cryptographic software...

Track and implement standards (ANSI, POSIX, parts of X/Open, etc.)

Work towards a very machine independent source tree. Support as many different systems and hardware as feasible.

Be as politics-free as possible; solutions should be decided on the basis of technical merit.

Do not let serious problems sit unsolved.

Provide a good cross compile/development platform.

Import external packages with minimal modifications - making upgrading much easier. Also to submit back to the developers any changes made.

Make a CDROM-based release approximately every six months, in particular to fund the project."

OpenBSD's goals are much easier to read and understand compared to FreeBSD, and more accurately reflect the project's purpose.

The NetBSD goals page says:

"The NetBSD Project provides a freely available and redistributable system that professionals, hobbyists, and researchers can use in whatever manner they wish."

Further:

"Generally speaking, the NetBSD Project provides a well designed, stable, and fast BSD system, avoids encumbering licenses, provides a portable system, which runs on many hardware platforms, interoperates well with other systems, [and] conforms to open systems standards as much as is practical."

NetBSD's goals appear to be the same as FreeBSD's.

Of these various sets of goals, I find OpenBSD's compelling and focused. I encourage readers to visit the OpenBSD goals page because they are explained more fully and link to further explanations.

I think it would be helpful to launch a new thread (similar to a recent one in freebsd-advocacy) publicly discussing the project's goals. I plan to post a link to this blog entry to the freebsd-advocacy mailing list and to BSDNews.com to encourage discussion. No single person can impose a set of goals on the FreeBSD project. Some sort of community consensus, ratified by the core team, would be useful. If you have comments, I recommend responding to my forthcoming posts on freebsd-advocacy and BSDNews.com.

Answering PHK's "Why Bother?" with FreeBSD Question

In the October issue of Daemon News, Poul-Henning Kamp asks "Why Bother?" He wants to know why people use FreeBSD when Linux gets most of the attention from users and vendors. He also wants to know why developers should continue to work on FreeBSD.

I will tailor my response for FreeBSD, as that is the BSD with which I am most familiar. Some of my arguments will apply to other variants. Some of my reasons even apply to other open source operating systems, like Linux. Few will apply to closed operating systems, least of which Windows.

1. FreeBSD is open source, with a business-friendly license. Being an open source, BSD-licensed operating system means I am free to modify the OS as I see fit and can continue to support and evolve any part of it, should the official developers decide to abandon any aspect of the project. I can use FreeBSD in commercial projects as long as I retain the copyright notice and disclaimer with the product.

2. FreeBSD is an integrated, complete product. FreeBSD isn't just a kernel supplemented by software. The developers provide easy (albeit time-consuming on slower machines) means to keep the system up-to-date and functioning properly. (Note: I will publish an article shortly that takes an in-depth look at keeping FreeBSD up-to-date.)

3. All FreeBSD source code is available via CVS. This feature allows users to quickly examine any aspect of the OS to determine if it will suit their needs. For example, I can look in a file like src/sys/dev/aac/aac_pci.c to see if it might support new hardware I'm considering purchasing. I can then checkout the complete CURRENT source or download a CURRENT .iso snapshot (via snapshots.se.freebsd.org or snapshots.jp.freebsd.org). This is perhaps the single greatest measurable advantage compared to closed-source software.

4. FreeBSD developers are accesible and analyze problems in context. I am constantly surprised to find so many developers posting answers to freebsd-questions and interacting helpfully with newsgroups like freebsd-stable and others. When a user posts a thoughtful question which demonstrates he's done some homework, he very frequently gets multiple responses that treat his problem within context of the entire OS. FreeBSD developers work within an operating system, not in isolation.

5. The FreeBSD ports tree offers over 12,000 applications, tailored for FreeBSD. When running FreeBSD, there's almost never a need to download and compile a tar.gz archive from a third party site. Instead, visit FreshPorts.org, see what's available, and then install either a precompiled package or build from source code within the ports system. When you see a program in the ports tree, you have some level of confidence that it will work on your system. When installed using the ports tree or as a precompiled package, you can now manage, track, and upgrade that application. (Note: I am planning a second article on this aspect of FreeBSD as well.)

6. FreeBSD's security features and track record are excellent. I can deploy a FreeBSD-based security appliance, with no ports listening other than OpenSSH, in less than 20 minutes (depending on CD-ROM speed). This system can defend itself without a network- or host-based firewall. FreeBSD has had an order of magnitude lower number of remote root vulnerabilities compared to Windows 2000, for example. Tools like FreeBSD-update, Portsnap, Portaudit and Portupgrade make keeping the system up-to-date easy. The VuXML project tracks issues not only in the OS but also the ports collection.

7. FreeBSD (as well as OpenBSD and NetBSD) Continue to Innovate. This is remarkable, despite the much smaller development community, user base, and lack of large commercial sponsorship. FreeBSD's SMP, network performance, TrustedBSD, and the ndis Windows driver wrapper are impressive. The OpenBSD project has brought us OpenSSH, Pf, OpenBGPD, Pfsync and CARP, and OpenNTPD. NetBSD introduced IPv6 and plenty of clean code that ends up in other projects. This is not to say that Linux or other UNIX derivatives don't innovate. I just amazed that a much smaller community has created these leading network services within a short period of time. I only expect better features out of FreeBSD's 5.x tree now that 5.3 has attained the STABLE mark.

There are seven reasons why I "bother" with FreeBSD. I would like to hear your comments, either via email or through postings to freebsd-advocacy or advocacy.daemonnews.org.

Sunday, November 28, 2004

Thoughts on the United States Air Force Computing Plans

As a former intelligence officer and computer network defender I was asked my thoughts on the US Air Force's new computing deal with Microsoft. In short, Microsoft will provide core server software, maintenance and upgrade support, and Dell will supply more than 525,000 Microsoft desktop Windows and Office software licenses to the Air Force.

From a business perspective, this is an important deal for Microsoft. For all of their seeming independence, the services tend to watch each other closely to see what technological advances are being considered or pursued. When the Navy began work on its Navy Marine Corp Intranet (NMCI), Air Force leaders scrambled to "catch up" to match the "progress" the Navy was assumed to be making. (NMCI has since produced mixed results for the Navy and financial woes for EDS, prime NMCI contractor.)

I have first-hand knowledge of the Air Force's response to NMCI. In the fall of 2000, the AFCERT sent me to Washington, DC to participate in the "redesign" of the Air Force enterprise network. Over a three week period we were expected to create plans to revamp the whole Air Force communications and security architecture. The new network was supposed to be running by June 2001. Obviously this did not happen, although it was the launch of the "One Air Force, One Network" campaign. The entire November 2000 (.pdf) issue of Intercom magazine was devoted to the project, while shorter articles explained the goals in brief.

The core of the project involved several key ideas:

- Server consolidation, particularly email
- Network access consolidation, with bases having links through their major commands
- An Air Force portal ("my.AF")

The Air Force is still following their "One Air Force, One Network" plans. You can see the Air Force CIO's September 2004 presentation (.ppt) reiterates these ideas, several of which I believe are valid. For example, it may not be necessary for each of the 100+ Air Force bases and operating locations to maintain their own connections to the Internet. It is certainly not necessary for each base to maintain its own Web, DNS, and mail servers, as these pieces of public infrastructure are one of the main ways external intruders can compromise Air Force assets.

Many reactions (e.g., Slashdot.org) to the new Microsoft/Air Force deal center on the standardization on Windows throughout the Air Force. Here I am disappointed. In the mid-1990s I used Solaris where it mattered, on servers and on intelligence projects. I remember using Windows for Workgroups 3.11 for office applications. Within the last five years Microsoft has replaced a lot of this infrastructure and handles core email and Web duties in many locations.

Regarding the implementation of a homogeneous infrastructure, I have a mixed opinion. Standardization can be a force for good when it eases system configuration, deployment, and patching. The question is, do the benefits of standardization outweigh the potential for thorough, widespread exploitation by a worm targeting standardized deployments? For years I've advocated deploying redundant architecture, running alternate operating systems, to mitigate this fate. If you want your Web site running IIS on Windows, have an Apache on UNIX backup ready.

I would have been pleased to see the Air Force adopt a thin client strategy for its desktop users, preferably based on Solaris or even a Linux distribution. I think the Air Force was too "corporate" to make such a radical step as to abandon its Microsoft desktop infrastructure. A TechWorld.com story noted that AF CIO "Gilligan acknowledged that in grappling with the patch-update issue, the Air Force had considered transitioning to open-source software but determined the transition costs would simply be too high." Ironically, that same article mentioned this:

"The Air Force endures about one network-based attack per week that successfully exploits new vulnerabilities, Gilligan said. 'There's some disruption and loss of capability,' he pointed out, noting that Air Force bases all over the world support the operations of the war in Afghanistan and Iraq. 'We're spending more money patching and fixing than buying software,' said Gilligan. It's not unusual for patching of vulnerabilities to take months to complete, he said."

So instead of taking a serious look at the root cause of its patching and exploitation costs (both financial and in mission impact), the Air Force sought a better deal from the vendor producing flawed software. This is sad. TechWorld's Ellen Messmer wrote "The US Air Force has had enough of Microsoft's security problems. But rather than switch to an alternative, it has struck a deal with CEO Steve Ballmer for a specially configured version of Windows." Will Microsoft sell this "special version" elsewhere, and if so, is the Air Force the guinea pig paying to develop this version?

Had the Air Force decided to break away from Microsoft, the other services would have definitely taken notice. In fact, corporate America would have taken notice. I hope vendors like Sun, IBM, and Red Hat step up their efforts to infiltrate the government space and introduce more secure products where it matters.

On a related note, the Air Force continues to lead the information warfare domain with its Advanced Course in Engineering (ACE) Cyber Security Boot Camp.

Friday, November 26, 2004

Five Ways Sguil is Different

On Wednesday I mentioned that a chapter from my book appeared in a new form at Informit.com. A snort-users reader asked how Sguil differed from ACID and BASE. In short, there are five reasons:

1. Sguil is a real-time interface to Snort alerts (and more).
2. Sguil is a Snort alert management system with integrated analyst accountability features.
3. Sguil offers growing alert handling capabilities.
4. Sguil is built to minimize "window management," "form management," and other non-analytical tasks.
5. Most importantly, Sguil is not limited to investigating events using Snort alert data alone.

To read explanations of each point, please see my response to the snort-users mailing list. You'll also find in that message three "features" that are not present in Sguil.

I should have mentioned that Sguil is the single tool most likely to provide analysts with the information they need to make a decision. With Sguil, a Snort alert is not the end of the investigation -- it's only the beginning.

Thursday, November 25, 2004

Metanetworks Claims "first wire-speed 10G Ethernet IDS/IPS product in the world"

While browsing the tcpdump-workers mailing list I came across a post describing the The Meta Traffic Processor PCI Card. This device "is a standard 32-bit/33MHz, PCI half-card with two copper Ethernet ports. The MTP appears to the host's operating system as a standard network interface card capable of enforcing from 600 to 1500 stateful policies to capture and/or block specific packets. The policies can be directly derived from public domain signatures or they can be completely user defined." These "policies" can be Snort rules or one day BPF filters.

The product appears to be the result of work done for a NSF grant. Metanetworks founder Livio Ricciulli described his project to NANOG in May 04.

I had heard from a .mil type that DoD was working with a vendor to run Snort-like applications in hardware. Perhaps this is what he meant?

I intend to contact Livio to see if his card works with FreeBSD.
Several recent Blog entries described ways to keep FreeBSD applications up-to-date. Based on my use of these tools, this is how I chose to update one of my servers this morning. First I updated the ports tree, INDEX-5, and INDEX.db:

cd /usr/ports
portsnap fetch
portsnap update
make fetchindex
portsdb -u

Next I checked to see which applications needed to be updated:

janney:/usr/ports# portversion -v -l "<"
bash-3.0.15 < needs updating (port has 3.0.16_1)
freebsd-update-1.6 < needs updating (port has 1.6_1)
sudo-1.6.8.1 < needs updating (port has 1.6.8.4)

I prefer to update applications by using precompiled packages provided by the FreeBSD team. Unfortunately, the packages-5-stable FTP site hasn't been updated since 15 Nov. I decided to press ahead and update these three packages on my own by building the upgrades from source.

I used portupgrade to (in the order of the switches shown), be verbose, update all packages whose versions are outdated, update packages that depend on the package being updated, update packages that the package being update considers dependencies, and create packages in /usr/ports/packages/All during the process:

janney:/usr/ports# portupgrade -varRp

We'll use the packages built during this process on server janney to update some of the packages on laptop orr.

When done I had three new packages in /usr/ports/packages/All:

janney:/usr/ports/packages/All# ls -alt
total 135276
drwxr-xr-x 21 root wheel 512 Nov 25 11:03 ..
-rw-r--r-- 1 root wheel 545101 Nov 25 11:03 bash-3.0.16_1.tbz
drwxr-xr-x 2 root wheel 1536 Nov 25 11:03 .
-rw-r--r-- 1 root wheel 10097 Nov 25 10:59 portsnap-0.2_1.tbz
-rw-r--r-- 1 root wheel 29219 Nov 25 10:58 freebsd-update-1.6_1.tbz
-rw-r--r-- 1 root wheel 99157 Nov 25 10:58 sudo-1.6.8.4.tbz

I expected bash, freebsd-update, and sudo to be there. These three were the packages identified by portversion as being out-of-date. A new portsnap package was created as part of the upgrade process for freebsd-update, since portsnap depends upon freebsd-update to function.

Now that I had these new packages, I turned to updating laptop orr. I followed the five steps I first did for janney, and a 'portversion -v -l "<"' to see what needed updating. Laptop orr has a lot more packages installed compared to janney. If I want to avoid updating packages on orr via building from source through the ports tree, I need to obtain updated packages either from the FreeBSD project or from a system that's built the same packages.

Standard FreeBSD systems do not seem to have a means to build packages without installing them. I believe OpenBSD has this capabilitity. Since I prefer to not maintain a "package builder" with ever application I need, I take a dual-pronged approach. First, I wait until the FreeBSD project provides updated packages. Second, for critical applications that tend to be installed on many systems, I create my own packages. This second approach is what this Blog entry is about.

Based on my update on server janney, I know I can provide updated bash, sudo, and freebsd-update packages for laptop orr. To do that I NFS mount /usr/ports/packages/All on janney to orr, then run portupgrade:

orr:/usr/ports# mount -t nfs janney:/usr/ports/packages/All /usr/ports/packages/All
orr:/usr/ports# portupgrade -varRPP
---> Session started at: Thu, 25 Nov 2004 11:26:02 -0500
** No need to upgrade 'tcpflow-0.21' (>= tcpflow-0.21). (specify -f to force)
---> Checking for the latest package of 'security/sudo'
---> Found a package of 'security/sudo': sudo-1.6.8.4.tbz (sudo-1.6.8.4)
---> Upgrade of security/sudo started at: Thu, 25 Nov 2004 11:26:09 -0500
---> Upgrading 'sudo-1.6.8.1' to 'sudo-1.6.8.4' (security/sudo) using a package
---> Updating dependency info
---> Uninstallation of sudo-1.6.8.1 started at: Thu, 25 Nov 2004 11:26:11 -0500
---> Fixing up dependencies before creating a package
---> Backing up the old version
---> Uninstalling the old version
---> Deinstalling 'sudo-1.6.8.1'
[Updating the pkgdb in /var/db/pkg ... - 167 packages found (-1 +0) (...) done]
---> Uninstallation of sudo-1.6.8.1 ended at: Thu, 25 Nov 2004 11:26:15 -0500 (consumed 00:00:03)
pkg_info: can't find package 'sudo-1.6.8.4.tbz' installed or in a file!
---> Installation of sudo-1.6.8.4 started at: Thu, 25 Nov 2004 11:26:15 -0500
---> Installing the new version via the package
Will not overwrite existing /usr/local/etc/sudoers file.
---> Removing temporary backup files
---> Installation of sudo-1.6.8.4 ended at: Thu, 25 Nov 2004 11:26:18 -0500 (consumed 00:00:03)
---> Cleaning out obsolete shared libraries
[Updating the pkgdb in /var/db/pkg ... - 168 packages found (-0 +1) . done]
---> Upgrade of security/sudo ended at: Thu, 25 Nov 2004 11:26:20 -0500 (consumed 00:00:10)
...truncated...
orr:/usr/ports# umount /usr/ports/packages/All

When done, bash, freebsd-update, and sudo were updated.

FreeBSD Ports Tree Breaks 12,000 Ports

Last night the FreeBSD ports tree broke the 12,000 mark. The tree has added about 2000 ports per year for the past four years. This graph shows the number of ports added per year since 1995.

I commend FreshPorts for providing such an excellent interface to the tree, and for keeping up with the growth in the number of applications available. FreshPorts recently integrated VuXML data, allowing users to visually see what port versions have a security issue. More details and information on other upgraded FreshPorts features are available here.

Wednesday, November 24, 2004

Using Portindex to Generate INDEX-5

Now that we've seen how to keep the ports tree up-to-date using tools like Portsnap, and seen how to generate an INDEX-5 file with 'make index', I'd like to offer an alternative INDEX-5 generation mechanism. Matthew Seaman's Portindex is a Perl tool replacing a similar application of the same name. The old version was pulled from the ports tree when the developer started acting strangely.

Here's the problem Portindex solves. If you use CVSup to update your ports tree, you will not have an INDEX-5 file. The INDEX-5 file included in CVS was reportedly always out-of-date, so you had to issue a command like 'portsdb -U' or 'make index' to create the INDEX-5 file. This was very time-consuming.

Portindex is a much faster alternative, once you've set up the system. Portindex is available in the ports tree as sysutils/p5-FreeBSD-Portindex , so installation is easy (as long as your ports tree is updated!) My installation is based on Matthew's excellent documentation at his site and in the Portindex man pages.

Once installed, you need to ininitalize the cache Portindex uses:

janney:/root# cache-init
Processing make describe output for path "/usr/ports": .........[1000].........F
reeBSD::Portindex::Tree::_scan_makefiles(): Can't open Makefile in /usr/ports/de
vel/jude_community -- No such file or directory at /usr/local/bin/cache-init lin
e 100
[2000].........[3000].........[4000].........[5000].........[6000].........[7000
].........[8000].........[9000].......FreeBSD::Portindex::Tree::_scan_makefiles(
): Can't open Makefile in /usr/ports/textproc/csv2txt -- No such file or directo
ry at /usr/local/bin/cache-init line 100
..[10000].........[11000].........<11993>

This took a long time -- hours -- on my dual PIII system. Once done, I had these new files:

janney:/var/db/portindex$ ls -al
total 9966
drwxrwxr-x 2 root operator 512 Nov 24 18:25 .
drwxr-xr-x 9 root wheel 512 Nov 24 18:22 ..
-rw-r--r-- 1 root operator 8192 Nov 24 21:10 __db.001
-rw-r--r-- 1 root operator 270336 Nov 24 21:10 __db.002
-rw-r--r-- 1 root operator 344064 Nov 24 21:10 __db.003
-rw-r----- 1 root operator 9617408 Nov 24 21:10 portindex-cache.db
-rw-r--r-- 1 root operator 25 Nov 24 18:25 portindex-timestamp

Next I had to update the ports tree and give Portindex the output in a form it could use:

janney:/root# script /tmp/cvsup.out cvsup -g -L2 -h cvsup2.freebsd.org
/usr/local/etc/ports-supfile
Script started, output file is /tmp/cvsup.out
Parsing supfile "/usr/local/etc/ports-supfile"
Connecting to cvsup2.freebsd.org
Connected to cvsup2.freebsd.org
Server software version: SNAP_16_1h
Negotiating file attribute support
Exchanging collection information
Establishing multiplexed-mode data connection
Running
Updating collection ports-all/cvs
Delete ports/INDEX-5
Edit ports/Tools/portbuild/scripts/buildscript
Add delta 1.17 2004.11.25.00.02.38 kris
...edited...
Checkout ports/x11-wm/ratpoison/files/ratpoison.desktop
Edit ports/x11-wm/ratpoison/pkg-plist
Add delta 1.6 2004.11.24.17.32.29 hq
Shutting down connection to server
Finished successfully

Script done, output file is /tmp/cvsup.out

Notice that at the end of this process, INDEX-5 is gone:

janney:/root# ls -al /usr/ports/IND*
-rw-r--r-- 1 root wheel 12250112 Nov 24 08:11 /usr/ports/INDEX.db

We saw earlier that CVS executed ' Delete ports/INDEX-5' when updating /usr/ports.

Now run 'cache-update' against the CVS output:

janney:/root# cache-update -i /tmp/cvsup.out
cache-update:1: Updating cached data for /usr/ports/audio/daapd
cache-update:2: Updating cached data for /usr/ports/audio/libsidplay2
cache-update:3: Updating cached data for /usr/ports/chinese/MT
cache-update:4: Updating cached data for /usr/ports/chinese/zhcon
cache-update:5: Updating cached data for /usr/ports/comms/efax-gtk
cache-update:6: Updating cached data for /usr/ports/comms/libticables
cache-update:7: Updating cached data for /usr/ports/comms/minicom
cache-update:8: Updating cached data for /usr/ports/databases/myodbc
cache-update:9: Updating cached data for /usr/ports/databases/slony1
cache-update:10: Updating cached data for /usr/ports/deskutils/hot-babe
cache-update:11: Updating cached data for /usr/ports/devel/bison
cache-update:12: Updating cached data for /usr/ports/devel/gettext
...edited...
cache-update:78: Updating cached data for /usr/ports/x11-wm/ratpoison
cache-update:79: Updating cached data for /usr/ports/x11/xextensions

Finally run 'portindex' to create INDEX-5:

janney:/root# portindex -o /usr/ports/INDEX-5
Accumulating dependency information: .........[1000].........[2000].........[300
0].........[4000].........[5000].........[6000].........[7000].........[8000]...
......[9000].........[10000].........[11000].........[12000]<12000>
Writing INDEX file: .........[1000].........[2000].........[3000].........[4000]
.........[5000].........[6000].........[7000].........[8000].........[9000].....
....[10000].........[11000].........[12000]<12000>
janney:/root# ls -al /usr/ports/IND*
-rw-r--r-- 1 root wheel 5959708 Nov 24 21:47 /usr/ports/INDEX-5
-rw-r--r-- 1 root wheel 12250112 Nov 24 08:11 /usr/ports/INDEX.db

I'm not sure if I will continue to use this method. As I just described it, you must use CVSup. Matthew offers 'find-updated' as a way to scan the /usr/ports tree and discover differences. I believe this could be paired with Portsnap to let Portsnap update the tree and Portindex update INDEX-5. That will probably be another Blog entry...

Installing Java on FreeBSD

In January I explained how I installed the java/jdk14 port on FreeBSD 5.2. Today I installed the java/jdk14 port on one 5.3 RELEASE system, janney, and then used packages built during the install process to install the JDK on my 5.3 RELEASE laptop, orr.

The FreeBSD project cannot distribute packages of the latest Java software due to Sun's licensing restrictions. (Really old 1.3.1 binary packages are offered from the FreeBSD Foundation.) I can create my own packages for personal use as I have read and accepted Sun's licenses. I cannot redistribute those packages for others to use.

First I ensured my ports tree was updated. Next I placed the necessary archives for the installation in /usr/ports/distfiles. These included:

bsd-jdk14-patches-6.tar.gz
j2re-1_4_2_06-linux-i586.bin
j2sdk-1_4_2-bin-scsl.zip
j2sdk-1_4_2-mozilla_headers-unix.zip
j2sdk-1_4_2-src-scsl.zip
j2sdk-1_4_2_06-linux-i586.bin

The first is available at EyesBeyond.com. The second is available as the J2SE v 1.4.2_06 JRE. The next three are available on the
Java 2 Platform, Standard Edition (J2SE) Sun Community Source License - Download
page. The last is posted as the J2SE v 1.4.2_06 SDK.

I also knew I had to have linprocfs already loaded. This meant that my /etc/fstab needed this entry:

linprocfs /compat/linux/proc linprocfs rw 0 0

I found that the best way to handle this was to add the entry to /etc/fstab and then reboot the system. I tried to manually follow these instructions to enable linprocfs:

kldload linprocfs
mount /compat/linux/proc

This did not work for me, since there was no /compat/linux/proc directory. I ended up creating a /compat/linux/proc directory by hand. (To be completely honest I lost track of how I ended up getting linprocfs to work, since my notes are blank at that point. I'd be interested in hearing if others had similar problems. I didn't seem to have any issues on 5.2 in January.)

Matthew Seaman's freebsd-java post explains that "in order to build JDK 1.4.2 you need a working JDK 1.4.x to compile everything. As things stand, that means at some point you have to use a Linux JDK to do an initial compilation."

A preview of what happens once you start the build process confirms Matthew's assertion:

===> jdk-1.4.2p6_6 depends on file: /usr/local/linux-sun-jdk1.4.2/bin/javac -
not found
===> Verifying install for /usr/local/linux-sun-jdk1.4.2/bin/javac in
/usr/ports/java/linux-sun-jdk14
======================================================================
Warning: This JDK may be unstable. You are advised to use the native
FreeBSD JDK, in ports/java/jdk14.

This Java VM will attempt to obtain some system information by
accessing files in linux's procfs. You must install the Linux
emulation procfs filesystem for this to work correctly. The JVM
will exhibit various problems otherwise. This can be accomplished
by adding the following line to your /etc/fstab file:

linprocfs /compat/linux/proc linprocfs rw 0 0

and then, as root, executing the commands:
kldload linprocfs
mount /compat/linux/proc

======================================================================
===> Vulnerability check disabled
===> Extracting for linux-sun-jdk-1.4.2.06
>> Checksum OK for j2sdk-1_4_2_06-linux-i586.bin.
...truncated...


Matthew continues: "Once you've compiled the JDK one time using the Linux emulation, you can discard the linux bits and use your native JDK to compile any updates... You can also take an installed native JDK 1.4.2 and create a package out of it." That's what we're going to do. Building java/jdk14 on host janney requires Linux emulation, but the package on laptop orr won't.

To start the build process I entered the /usr/ports/java/jdk14 directory and ran this command:

janney:/usr/ports/java/jdk14# make package-recursive DISABLE_VULNERABILITIES=true

The 'package-recursive' directive tells the make process to create the java/jdk14 package, as well as packages for all of the java/jdk14 dependencies. When the process is finished, you'll have packages (.tbz files) in /usr/ports/packages/All.

The 'DISABLE_VULNERABILITIES=true' directive disables vulnerability checking provided via the portaudit tool. I generally do not advocate this step because you are potentially installing software with vulnerabilities. For example, I originally executed 'make package-recursive' and had this result:

===> jdk-1.4.2p6_6 depends on file: /usr/X11R6/lib/libXm.so - not found
===> Verifying install for /usr/X11R6/lib/libXm.so in /usr/ports/x11-toolkits
/open-motif
===> open-motif-2.2.3 has known vulnerabilities:
>> xpm -- image decoding vulnerabilities.
Reference: http://www.FreeBSD.org/ports/portaudit/ef253f8b-0727-11d9-b45d-000c41e2cdad.html
>> Please update your ports tree and try again.
*** Error code 1

Stop in /usr/ports/x11-toolkits/open-motif.
*** Error code 1

Stop in /usr/ports/java/jdk14.

A visit to www.freshports.org/x11-toolkits/open-motif shows a new feature -- the "skull icon" indicating a security vulnerability in the desginated version of x11-toolkits/open-motif. Although OpenMotif 2.2.4, patching the hole, is available, it has not been integrated into the FreeBSD ports tree. According to the portaudit and VuXML reports, X11R6.8.1 fixes the problem. A look through the freebsd-x11 mailing list shows that a port of xorg 6.8.1 is in the works but not finished.

There's a second security issue to consider. Sun released an advisory warning customers to not use the "SDK and JRE [versions] 1.4.2_05 and earlier, all 1.4.1 and 1.4.0 releases, and [version] 1.3.1_12 and earlier." This follows a post to full-disclosure, publicizing the issue. FreeBSD Java patch set developer Greg Lewis committed an update marking the java/jdk14 port "forbidden" when building the browser plug-in because of this vulnerability. In fact, you'd see this warning even if yo do disable vulnerability checks:

===> jdk-1.4.2p6_6 is forbidden: Vulnerabilities in the browser plugin.

In an email exchange Greg assured me that "vulnerability exists in the current patchset and will do so until we find a fix for it." Besides invoking 'DISABLE_VULNERABILITIES=true', one could also build java.jdk14 with 'MINIMAL=true', as that directive says 'don't build/install mozilla java plugin, javaws and JDK demos'.

Since I was deploying Java to support some applications, and not run in a browser, I wasn't worried about reading either malicious images (affecting xpm) or applets (affecting the Java JRE).

When the build process is done, execute 'make install' to install java/jdk14.

If you're sucessful, you'll find jdk-1.4.2p6_6.tbz plus other packages in /usr/ports/packages/All. I chose to install these on my laptop my NFS mounting /usr/ports from build server janney as /mnt on laptop orr:

orr:/root# mount -t nfs janney:/usr/ports /mnt
orr:/root# mount | grep janney
janney:/usr/ports on /mnt (nfs)

Now I could install the package and its dependencies on orr:

orr:/root# pkg_add -v /mnt/packages/All/jdk-1.4.2p6_6.tbz

A successful installation can be checked with this command:

orr:/root# java -version
java version "1.4.2-p6"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2-p6-root_24_nov_2004_10_37)
Java HotSpot(TM) Client VM (build 1.4.2-p6-root_24_nov_2004_10_37, mixed mode)

I like this method of building time-intensive, complicated ports elsewhere (on more robust systems) and then installing them as packages via NFS on end-user systems. Had a precomplied package been available from the FreeBSD project or elsewhere, I would have used that. (That's how I installed OpenOffice.org, another bear to compile from source.) This system works well when you really do need to "roll your own." Thankfully the FreeBSD ports tree simplifies that process.

CERT/CC Publishes Principles of Survivability and Information Assurance

CERT/CC just published ten Principles of Survivability and Information Assurance. They are:

1. Survivability is an enterprise-wide concern.

2. Everything is data.

3. Not all data is of equal value to the enterprise – risk must be managed.

4. Information assurance policy governs actions.

5. Identification of users, computer systems, and network infrastructure components is critical.

6. Survivable Functional Units (SFUs) are a helpful way to think about an enterprise’s networks.

7. Security Knowledge in Practice (SKiP) provides a structured approach.

8. The road map guides implementation choices (all technology is not equal).

9. Challenge assumptions to understand risk. (Think like an intruder.)

10. Communication skill is critical to reach all constituencies.

Some of these principles are backed up by their own papers or CERT practices. They are a good starting point to measure an organization's overall security posture.

Informit.com Publishes Sguil Chapter

My buddy Keith McCammon told me he found Informit.com serving up chapter 10 from The Tao of Network Security Monitoring as Why Sguil Is the Best Option for Network Security Monitoring Data. Controversial, and not my doing. :) Still, the whole chapter is there in browsable HTML, along with embedded screenshots. We are still testing Sguil 0.5.3 but expect to release it soon.

Using FreeBSD Update to Patch FreeBSD

When the FreeBSD Security team released an advisory for fetch(1), I knew I could turn to Colin Percival's FreeBSD Update for binary security upgrades.

Installation is simple. Here's how to installing via package:

pkg_add -vr freebsd-update
mkdir /usr/local/freebsd-update
cp /usr/local/etc/freebsd-update.conf.sample /usr/local/etc/freebsd-update.conf

Here is how FreeBSD Update patched the fetch(1) vulnerability:

orr:/root# freebsd-update fetch
Fetching public key...
Fetching updates signature...
Fetching updates...
Fetching hash list signature...
Fetching hash list...
Examining local system...
Fetching updates...
/usr/bin/fetch...
Updates fetched

To install these updates, run: '/usr/local/sbin/freebsd-update install'
orr:/root# freebsd-update install
Backing up /usr/bin/fetch...
Installing new /usr/bin/fetch...


That's it. I didn't need to CVSup to STABLE or manually patch the fetch(1) binary. FreeBSD Update handled it, and with the change being to userland, no reboot is necessary.
The traditional means to update the FreeBSD ports tree involves using CVSup in a manner described well by Dru Lavigne. After the sysutils/cvsup
port is installed, and ports-supfile modified to point to a real CVSup server (e.g. *default host=cvsup8.FreeBSD.org'), you run commands like this:

cvsup -g -L 2 /usr/local/etc/ports-supfile
portsdb -uU

The first command updates the ports tree in /usr/ports. The second command creates or updates the INDEX file (via the 'U') and then creates or updates the INDEX.db file (via the 'u') from the INDEX.

There is now an alternative to CVSup available in the ports tree: Portsnap, written by FreeBSD-update author Colin Percival. Installation via packages is easy:

orr:/root# setenv PACKAGESITE ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/Latest/
orr:/root# pkg_add -vr portsnap
...edited...
Package portsnap-0.2_1 registered in /var/db/pkg/portsnap-0.2_1

Before you can use portsnap, you will have to create an update configuration
file specifying the server from which to fetch snapshots and the sha1 hash
of the RSA public key which is trusted to sign the snapshots.

A sample configuration file has been installed in

/usr/local/etc/portsnap.conf.sample

which will fetch snapshots built and signed by the author. If you want to
use these updates, copy that file to

/usr/local/etc/portsnap.conf

otherwise, create that file as appropriate.

orr:/root# cp /usr/local/etc/portsnap.conf.sample /usr/local/etc/portsnap.conf
orr:/root# mkdir /usr/local/portsnap

Once installed, see what options Portsnap offers:

orr:/root# portsnap -h
usage: portsnap [options] command [URL]

Options:
-d workdir -- Store working files in workdir
(default: /usr/local/portsnap/)
-f conffile -- Read configuration options from conffile
(default: /usr/local/etc/portsnap.conf)
-k KEY -- Trust an RSA key with SHA1 hash of KEY.
-p portsdir -- Location of uncompressed ports tree
(default: /usr/ports/)
URL -- Fetch updates from given URL.
Commands:
fetch -- Fetch a compressed snapshot of the ports tree,
or update an existing snapshot.
cron -- Sleep rand(3600) seconds, and then fetch updates.
extract -- Extract snapshot of ports tree, replacing existing
files and directories.
update -- Update ports tree to match current snapshot, replacing
files and directories which have changed.

Here is a sample run:

orr:/root# portsnap fetch
Fetching public key... done.
Fetching snapshot tag... done.
Fetching snapshot generated at Sun Oct 24 14:21:27 EDT 2004:
c2246a9802a7155d099dd37fdfcbbc0f77b5864a.tgz 3% of 31 MB 192 kBps
Extracting snapshot... done.
Verifying snapshot integrity... done.
Fetching updated snapshot tag... done.
Updating from Sun Oct 24 14:21:27 EDT 2004 to Wed Nov 24 10:04:58 EST 2004.
Attempting to generate index via delta compression... success.
Generating list of updates needed... 2401 files or ports need to be updated.
Attempting to fetch 2210 patches... 2209 fetched.
Attempting to apply patches... done.
Attempting to fetch 192 new files or ports... done.

orr:/root# portsnap extract
/usr/ports/.cvsignore
/usr/ports/CHANGES
/usr/ports/LEGAL
/usr/ports/MOVED
/usr/ports/Makefile
/usr/ports/Mk/bsd.autotools.mk
/usr/ports/Mk/bsd.emacs.mk
/usr/ports/Mk/bsd.gnome.mk
...edited...
/usr/ports/x11/yalias/
/usr/ports/x11/yelp/
/usr/ports/x11/zenity/

After the initial 'extract', future updates just use 'update':

orr:/root# portsnap fetch
Fetching updated snapshot tag... done.
Updating from Wed Nov 24 15:36:20 EST 2004 to Wed Nov 24 17:36:13 EST 2004.
Attempting to generate index via delta compression... success.
Generating list of updates needed... 8 files or ports need to be updated.
Attempting to fetch 8 patches... 8 fetched.
Attempting to apply patches... done.
Attempting to fetch 0 new files or ports... done.
orr:/root# portsnap update
Removing old files and directories... done.
Extracting new files:
/usr/ports/chinese/zhcon/
/usr/ports/comms/minicom/
/usr/ports/devel/gettext/
/usr/ports/games/wolfpack/
/usr/ports/math/ndiff/
/usr/ports/science/mmtk/
/usr/ports/textproc/libxml2/
/usr/ports/textproc/py-docutils/

That's it. Now, the INDEX-5 file hasn't been updated. A great way to do that is to use 'make fetchindex'. Here's a subsequent run of Portsnap followed by 'make index':

orr:/root# portsnap fetch
Fetching updated snapshot tag... done.
Updating from Wed Nov 24 17:36:13 EST 2004 to Wed Nov 24 19:36:13 EST 2004.
Attempting to generate index via delta compression... success.
Generating list of updates needed... 20 files or ports need to be updated.
Attempting to fetch 17 patches... 17 fetched.
Attempting to apply patches... done.
Attempting to fetch 3 new files or ports... done.
orr:/root# portsnap update
Removing old files and directories... done.
Extracting new files:
/usr/ports/Tools/portbuild/
/usr/ports/audio/libsidplay2/
/usr/ports/chinese/MT/
/usr/ports/comms/libticables/
/usr/ports/devel/bison/
/usr/ports/devel/libmcve/
/usr/ports/devel/libtifiles/
/usr/ports/java/linux-blackdown-jdk14/
/usr/ports/lang/gcc28/
/usr/ports/lang/nickle/
/usr/ports/net/jit/
/usr/ports/net/libicq2000/
/usr/ports/www/opera/
/usr/ports/x11-themes/Makefile
/usr/ports/x11-themes/kde-icons-cezanne/
/usr/ports/x11-themes/kde-icons-gartoon-blue-svg/
/usr/ports/x11-themes/kde-icons-gartoon-svg/
/usr/ports/x11-themes/kde-icons-noia/
/usr/ports/x11-themes/kde-icons-sparkling/
/usr/ports/x11/xextensions/
orr:/root# ls -al /usr/ports/IND*
-rw-r--r-- 1 root wheel 5089899 Apr 30 2004 /usr/ports/INDEX
-rw-r--r-- 1 root wheel 6018417 Nov 19 21:02 /usr/ports/INDEX-5
-rw-r--r-- 1 root wheel 12334080 Nov 19 22:22 /usr/ports/INDEX.db
orr:/root# cd /usr/ports
orr:/usr/ports# make fetchindex
INDEX-5.bz2 100% of 589 kB 159 kBps
orr:/usr/ports# ls -al /usr/ports/IND*
-rw-r--r-- 1 root wheel 5089899 Apr 30 2004 /usr/ports/INDEX
-rw-r--r-- 1 root wheel 6040505 Nov 24 21:03 /usr/ports/INDEX-5
-rw-r--r-- 1 root wheel 12334080 Nov 19 22:22 /usr/ports/INDEX.db

Notice how Portsnap did not affect INDEX-5, but 'make fetchindex' brings a new INDEX-5. To update INDEX.db, run 'portsdb -u':

orr:/usr/ports# portsdb -u
[Updating the portsdb in /usr/ports ... - 11999 port entries found ...
......1000.........2000.........3000.........4000.........5000
.........6000.........7000.........8000.........9000.........10000.....
....11000......... ..... done]
orr:/usr/ports# ls -al /usr/ports/IND*
-rw-r--r-- 1 root wheel 5089899 Apr 30 2004 /usr/ports/INDEX
-rw-r--r-- 1 root wheel 6040505 Nov 24 21:03 /usr/ports/INDEX-5
-rw-r--r-- 1 root wheel 12364800 Nov 24 21:21 /usr/ports/INDEX.db

Notice that we're only one port away from 12,000!

Tuesday, November 23, 2004

Prof Kerr on KeyKatcher Case

I always enjoy reading Professor Orin Kerr's Computer Crime Case Updates. This week he comments on the dismissed wiretapping case mentioned by SecurityFocus.com and Slashdot. Although some commentary from the likes of Slashdot is helpful, I prefer reading the opinions of a Harvard Law graduate and former Supreme Court clerk.

The case is simple: does use of a keystroke logger constitute a wiretap? The judge in the case said no. I agree with Prof Kerr's assessment that the opinion is wrong. If someone listens in on a phone between the handset and the base unit, it's still a wiretap. It's no different if someone collects keystrokes using a device between a keyboard and CPU.

However, I disagree with Prof Kerr's reasoning concerning interstate commerce. Plenty of judges disagree with me, but I don't think connecting to the Internet makes a system automatically engaged in "interstate commerce." I think the use of the so-called Interstate Commerce Clause allows Congress to pass laws that far exceed their true Constitutional mandate.

If you've never read Prof Kerr's opinions, I recommend you browse his mailing list archives. They're incredibly enlightening.

Kudos for Proper Incident Handling at The Register

The UK-based news site The Register was victimized by an advertisement provider, Falk AG, beginning Saturday. The ads served by Falk AG were carriers for the Bofra worm, which uses a buffer overflow in FRAME, IFRAME, and EMBED elements of pre-XP SP2 Internet Explorer.

The Register promptly issued a warning on Sunday morning, followed by a statement on restoration of service this morning. The Register estimates the number of visitors who could have been affected by this event, which is a good way to scope the extent of the incident.

Falk AG has also owned up to the incident, although its wording leaves a little to be desired. From the company's statement:

"Early Saturday morning (20.11.2004) an unauthorized individual exploited a weakness in a load balancer on the European AdSolution network. The purpose of the exploit was to establish a redirect to malicious code through a javascript component of Falk’s ad delivery... Unauthorized access was possible only as a result the intentional exploitation of a weak point of a network load balancer located in the EU datacenter. Once accessed, the individual was able to modify a configuration which forced the redirect to the malicious code."

I like the mention of a "weakness" and a "weak point." That sounds like press-speak for misconfiguration, or unpatched vulnerability. Although Falk has many clients, on Dutch news site Nu.nl has reported on the event, along with The Reg.

According to this site, Falk has a history of serving up Trojaned ads. Maybe that will give me some traffic to inspect for my next book?

Sunday, November 21, 2004

OpenOffice.org 1.1.3 Packages for FreeBSD 5.3 RELEASE

Back in January I described installing OpenOffice.org 1.1.0 on FreeBSD 5.2 RELEASE. I used packages from the FreeBSD OpenOffice Porting Team. I am happy to report I installed OpenOffice.org 1.1.3 on FreeBSD 5.3 RELEASE using packages hosted at http://oootranslation.services.openoffice.org/pub/OpenOffice.org/ooomisc/FreeBSD/.

The FreeBSD project does not maintain precompiled packages for OpenOffice.org (OOo), presumably because it takes too long for the build cluster to create the packages from source. Therefore, one cannot do 'pkg_add -vr openoffice' and expect to download packages from a FreeBSD FTP site.

This means that you usually download the appropriate OOo package from the aforementioned Web site and execute 'pkg add -v OOo_1.1.3_FreeBSD53Intel_install.tbz' . If you are missing a dependency, however, you must add it manually. Consider what happened when I took these actions:

pkg_add -v OOo_1.1.3_FreeBSD53Intel_install.tbz
Requested space: 274391836 bytes, free space: 4726294528 bytes in /var/tmp/instmp.vf16XV
Package 'openoffice-1.1.3' depends on 'freetype2-2.1.7_3' with 'print/freetype2'
origin.
- already installed.
Package 'openoffice-1.1.3' depends on 'expat-1.95.8' with 'textproc/expat2' origin.
- already installed.
...edited...
Package 'openoffice-1.1.3' depends on 'ORBit-0.5.17_2' with 'devel/ORBit' origin.
pkg_add: could not find package ORBit-0.5.17_2 !
pkg_add: 1 package addition(s) failed

Because ORBit was not installed, I added it manually. Then I was able to install the OOo package:

pkg_add -vr ORBit
...edited...
pkg_add -v OOo_1.1.3_FreeBSD53Intel_install.tbz
extract: Package name is openoffice-1.1.3
extract: CWD to /usr/local
extract: /usr/local/bin/openoffice-1.1.3
extract: /usr/local/bin/openoffice-1.1.3-sagenda
extract: /usr/local/bin/openoffice-1.1.3-scalc
...edited...
OpenOffice.org Build 1.1.3 Personal Install How-To

Written by: Martin Blapp

OpenOffice.org 1.1.3 will soon been installed in
${PREFIX}/OpenOffice.org-1.1.3

1 User installation
-------------------

Just type "openoffice" after you have successfully
installed the package. If there is no installed
OO.org dir in hour homedir, you'll be prompted to
install some files and choose a installed JDK.
The setup installs a "OpenOffice.org1.1.3" folder
in your homedir.

If the setup tells you there is already an installed
version, you may look at the file ".sversionrc" in
your homedir. In this file OpenOffice and StarOffice
have both a line for each version which is installed.
After removing the problematic line you should be able to
install again.

2 Start OO.org
--------------

There are some wrappers installed for fast startup.
Add "${PREFIX}/bin/" to your PATH and you will be able
to use them.

${PREFIX}/bin/openoffice-1.1.3
${PREFIX}/bin/openoffice-1.1.3-sagenda
${PREFIX}/bin/openoffice-1.1.3-scalc
${PREFIX}/bin/openoffice-1.1.3-sdraw
${PREFIX}/bin/openoffice-1.1.3-setup
${PREFIX}/bin/openoffice-1.1.3-sfax
${PREFIX}/bin/openoffice-1.1.3-simpress
${PREFIX}/bin/openoffice-1.1.3-spadmin
${PREFIX}/bin/openoffice-1.1.3-sweb
${PREFIX}/bin/openoffice-1.1.3-swriter

When done I ran 'openoffice-1.1.3', which invoked a setup program. When it was finished, I was able to start OOo with the same 'openoffice-1.1.3' command.

Saturday, November 20, 2004

Fixing Weird Behavior after Thunderbird and Mozilla Upgrades

FreeBSD 5.3 RELEASE shipped with thunderbird-0.7.3_1.tbz and firefox-0.9.3_1.tbz. Newer precompiled packages are available, so I upgraded to thunderbird-0.9_2 and firefox-1.0,1.

When I started Thunderbird, I found I had no mail in any of my folders. My default folder languages had also changed to Arabic. After searching the Web, I found this forum which suggested deleting the compreg.dat file in the .thunderbird/default.XXX directory. Sure enough, after renaming the file, a restarted Thunderbird performed as it should have.

When I started Firefox, I found that middle-clicking on links launched a new, yet empty, tab. I also couldn't use my arrow keys to navigate the address bar. I decided to move the compreg.dat file in the .mozilla/firefox/default.XXX directory. After restarting Firefox, it too worked fine.

Friday, November 19, 2004

Upgrading FreeBSD Packages with Portupgrade and without CVSup

freebsd.png" align=left>As much as I like FreeBSD's ports system, I dislike keeping the tree up-to-date using CVSup. I haven't yet tried Colin Percival's portsnap tool, which uses HTTP to distribute compressed and cryptographically signed snapshots of the ports tree. This is a great alternative, but I try to avoid compiling from source using the ports tree when possible. I run several very generic and slow systems, and I'd much rather install and upgrade software using precomplied packages.

Portupgrade is the tool of choice to update software on FreeBSD. Portupgrade is powerful because it can (far more often than not) resolve dependencies to upgrade software en masse. It doesn't matter if an application is installed via ports or precompiled packages. Once installed, FreeBSD treats it like a package using tools like pkg_info. When invoked with the -PP switch, portupgrade can be told to only upgrade installed software by using packages. This avoids having to compile upgraded software in order to install it.

Most people still say the ports tree must be updated to let portupgrade know the newest version of the port available. So, even when upgrading using packages, it seems a CVSup of the ports tree, or an equivalent process, is necessary. Or is it?

I recently started thinking about alternatives to updating the whole ports tree. There's been some discussions regarding the ports system on the freebsd-ports mailing list. This post announced the removal of the INDEX and INDEX-5 files from CVS. These files (INDEX for the 4.x tree and INDEX-5 for the 5.x tree) list information like the following excerpt for nmap:

nmap-3.75|/usr/ports/security/nmap|/usr/local|Port scanning utility for large
networks|/usr/ports/security/nmap/pkg-descr|eik@FreeBSD.org|security ipv6|openssl-
0.9.7e_1 pcre-5.0|openssl-0.9.7e_1 pcre-5.0|http://www.insecure.org/nmap/|||

This entry describes the nmap port, its maintainer, dependencies, and so on. You won't find any INDEX files in CVS at cvsweb.freebsd.org, for example. (Here's the entry in the CHANGES file also announcing the INDEX removals.) The maintainers thought that since the INDEX files kept in CVS were always out-of-date, it wasn't necessary to keep them there. INDEX files are available elsewhere, however, either by running 'make index' in /usr/ports or by downloading them. The latter option is going to help us, as the former takes a lot of time using the traditional methods currently available.

It turns out that you can completely avoid CVSup and upgrading the ports tree if you want to work strictly with precompiled packages. Here's how to do it on a FreeBSD 5.3 RELEASE system with several applications already installed via precompiled 5.3 RELEASE packages.

When I started I only had the INDEX-5 file that accompanied the ports collected installed with FreeBSD 5.3 RELEASE:

janney:/root# ls -al /usr/ports/IND*
-rw-r--r-- 1 root wheel 5798559 Nov 4 18:27 /usr/ports/INDEX-5

First I installed portupgrade:

janney:/root# pkg_add -vr portupgrade
looking up ftp.freebsd.org
connecting to ftp.freebsd.org:21
setting passive mode
opening data connection
initiating transfer
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-5.3-release/Lates
t/portupgrade.tbz...x +CONTENTS
x +COMMENT
x +DESC
x +MTREE_DIRS
x man/man1/pkg_deinstall.1.gz
...truncated...

Once portupgrade was installed I ran portversion to see how portupgrade saw my installed packages. Running portversion also built the package database, /var/db/pkg/pkgdb.db.

janney:/root# portversion -v
[Updating the pkgdb in /var/db/pkg ... - 48 packages found (
-0 +1) . done]
[Updating the portsdb in /usr/ports ... - 11735 port entries
found .........1000.........2000.........3000.........4000.........5000........
.6000.........7000.........8000.........9000.........10000.........11000.......
..... done]
bash-3.0_5 = up-to-date with port
bitstream-vera-1.10 = up-to-date with port
boxtools-0.65.0 = up-to-date with port
cdrtools-2.0.3_4 = up-to-date with port
cmdwatch-0.2.0 = up-to-date with port
...truncated...

When the process completed I had a new INDEX.db file in /usr/ports, along with the original INDEX-5:

janney:/root# ls -al /usr/ports/IND*
-rw-r--r-- 1 root wheel 5798559 Nov 4 18:27 /usr/ports/INDEX-5
-rw-r--r-- 1 root wheel 11915264 Nov 19 23:16 /usr/ports/INDEX.db

Now I needed to update that old, original INDEX-5 file. As far as portversion was concerned, all of my installed packages were up-to-date. Unfortunately, this was not true. For example, a look at ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/ shows that shells/bash-3.0.15.tbz is available. I'm using bash-3.0_5, so there's a newer precompiled package ready for me to upgrade.

The key to this process is twofold. First, obtain a new INDEX-5 file by running the following:

janney:/usr/ports# make fetchindex
INDEX-5 0% of 5877 kB 0 Bps

When 'make fetchindex' is done, the INDEX-5 is updated:

janney:/usr/ports# ls -al IND*
-rw-r--r-- 1 root wheel 6018417 Nov 19 23:05 INDEX-5
-rw-r--r-- 1 root wheel 11915264 Nov 19 23:16 INDEX.db

Second, force an update of the INDEX.db file:

janney:/usr/ports# portsdb -uf
[Updating the portsdb in /usr/ports ... - 11972 port entries
found .........1000.........2000.........3000.........4000.........5000........
.6000.........7000.........8000.........9000.........10000.........11000........
. ..... done]

Now look at INDEX.db:

janney:/usr/ports# ls -al IND*
-rw-r--r-- 1 root wheel 6018417 Nov 19 23:05 INDEX-5
-rw-r--r-- 1 root wheel 12334080 Nov 19 23:54 INDEX.db

Running portversion again shows several ports requiring updates. Only bash is visible below:

janney:/usr/ports# portversion -v
bash-3.0_5 < needs updating (port has 3.0.16_1)
bitstream-vera-1.10 = up-to-date with port
boxtools-0.65.0 = up-to-date with port
cdrtools-2.0.3_4 = up-to-date with port
...truncated...

Now the INDEX-5 file indicates that the newest version of bash in the ports tree is 3.0.16_1, but the precompiled package is only 3.0.15. This is acceptable (as long as we are not addressing security issues), so we will update bash from 3.0_5 to 3.0.15.

The next step is to tell portupgrade where to find precompiled packages:

janney:/usr/ports# setenv PACKAGESITE ftp://ftp2.freebsd.org/pub/FreeBSD/ports/
i386/packages-5-stable/All/

Finally we are ready to begin upgrading packages with portupgrade:

janney:/usr/ports# portupgrade -varRPP
---> Session started at: Fri, 19 Nov 2004 23:55:03 -0500
** No need to upgrade 'python-2.3.4_2' (>= python-2.3.4_2). (specify -f to force)
...edited...
---> Checking for the latest package of 'shells/bash'
---> Fetching the package(s) for 'bash-3.0.16' (shells/bash)
---> Fetching bash-3.0.16
++ Will try the following sites in the order named:
ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/
---> Invoking a command: /usr/bin/fetch -o '/var/tmp/bash-3.0.16.tbz' 'ftp://ft
p2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/All/bash-3.0.16.tbz'
fetch: ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/All/bash-
3.0.16.tbz: File unavailable (e.g., file not found, no access)
** The command returned a non-zero exit status: 1
** Failed to fetch ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stab
le/All/bash-3.0.16.tbz
---> Invoking a command: /usr/bin/fetch -o '/var/tmp/bash-3.0.16.tgz' 'ftp://ft
p2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/All/bash-3.0.16.tgz'
fetch: ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/All/bash-
3.0.16.tgz: File unavailable (e.g., file not found, no access)
** The command returned a non-zero exit status: 1
** Failed to fetch ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stab
le/All/bash-3.0.16.tgz
** Failed to fetch bash-3.0.16
---> Listing the results (+:done / -:ignored / *:skipped / !:failed)
! bash-3.0.16 (fetch error)
---> Packages processed: 0 done, 0 ignored, 0 skipped and 1 failed
---> Fetching the latest package(s) for 'bash' (shells/bash)
---> Fetching bash
++ Will try the following sites in the order named:
ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/
---> Invoking a command: /usr/bin/fetch -o '/var/tmp/bash.tbz' 'ftp://ftp2.free
bsd.org/pub/FreeBSD/ports/i386/packages-5-stable/Latest/bash.tbz'
/var/tmp/bash.tbz 0% of 532 kB
---> Downloaded as bash.tbz
---> Identifying the package /var/tmp/bash.tbz
---> Saved as /usr/ports/packages/All/bash-3.0.15.tbz
---> Skipping libiconv-1.9.2_1 (already installed)
---> Skipping gettext-0.13.1_1 (already installed)
---> Listing the results (+:done / -:ignored / *:skipped / !:failed)
+ bash@
- libiconv-1.9.2_1
- gettext-0.13.1_1
---> Packages processed: 1 done, 2 ignored, 0 skipped and 0 failed
---> Found a package of 'shells/bash': bash-3.0.15.tbz (bash-3.0.15)
---> Located a package version 3.0.15 (bash-3.0.15.tbz)
---> Using it anyway although it is not the latest version (3.0.16), since -PP/
--use-packages-only is specified
---> Upgrade of shells/bash started at: Sat, 20 Nov 2004 00:00:52 -0500
---> Upgrading 'bash-3.0_5' to 'bash-3.0.15' (shells/bash) using a package
---> Updating dependency info
---> Uninstallation of bash-3.0_5 started at: Sat, 20 Nov 2004 00:00:53 -0500
---> Fixing up dependencies before creating a package
---> Backing up the old version
---> Uninstalling the old version
---> Deinstalling 'bash-3.0_5'
[Updating the pkgdb in /var/db/pkg ... - 47 packages found (
-1 +0) (...) done]
---> Uninstallation of bash-3.0_5 ended at: Sat, 20 Nov 2004 00:01:01 -0500 (co
nsumed 00:00:07)
pkg_info: can't find package 'bash-3.0.15.tbz' installed or in a file!
---> Installation of bash-3.0.15 started at: Sat, 20 Nov 2004 00:01:01 -0500
---> Installing the new version via the package
---> Removing temporary backup files
---> Installation of bash-3.0.15 ended at: Sat, 20 Nov 2004 00:01:06 -0500 (con
sumed 00:00:04)
---> Cleaning out obsolete shared libraries
[Updating the pkgdb in /var/db/pkg ... - 48 packages found (
-0 +1) . done]
---> Upgrade of shells/bash ended at: Sat, 20 Nov 2004 00:01:08 -0500 (consumed
00:00:15)
** No need to upgrade 'xorg-fonts-miscbitmaps-6.7.0' (>= xorg-fonts-miscbitmaps-
6.7.0). (specify -f to force)
...truncated...

In included the whole listing here to show what happened when portupgrade couldn't find the version of bash it wanted, namely bash-3.0.16, at ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/All. It knew to look for bash.tbz at ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/Latest however. That location is a symlink to the bash package available:

lrwxr-xr-x 1 110 0 22 Nov 15 08:09 bash.tbz -> ../All/bash-3.0.15.tbz

To save some hassle, in the future I will specify the PACKAGESITE like this:

janney:/usr/ports# setenv PACKAGESITE ftp://ftp2.freebsd.org/pub/FreeBSD/ports/
i386/packages-5-stable/All/

When done, all of the ports which could be updated via precomplied package, were indeed updated. For example, here is bash:

janney:/root# portversion -v bash
bash-3.0.15 < needs updating (port has 3.0.16_1)

Since precompiled packages are uploaded to the FTP servers weekly, I intend to repeat this process shortly after the next set of packages appears.

I also intend to keep an eye on Matthew Seaman's new portindex tool. This follows the absolutely bizarre removal of the original portindex from the ports tree in September. Matthew's portindex allows fast generation of INDEX and INDEX-5, replacing the slow 'make index' or 'portsb -uU' processes. The new portindex program, however, builds from an updated ports tree, which my package-centric process avoids.

For those wishing to use ports, however, the combination of portindex with portsnap will be powerful. Those wanting to stick with packages for slower or generic deployments might like the system explained here.

Great Thread on Network Performance Troubleshooting

Now that FreeBSD 5.3 has arrived, users are trying to determine if any performance issues are caused by their hardware, OS, or applications. There's a great freebsd-stable thread discussing a user's attempt to improve NFS performance. One of Robert Watson's comments is especially useful, since he spells out five steps to troubleshoot network performance:

"I think the first thing you want to do is to try and determine whether the problem is a link layer problem, network layer problem, or application (file sharing) layer problem. Here's where I'd start looking:

(1) I'd first off check that there wasn't a serious interrupt problem on the box, which is often triggered by ACPI problems. Get the box to be as idle as possible, and then use vmstat -i or stat -vmstat to see if anything is spewing interrupts.

(2) Confirm that your hardware is capable of the desired rates: typically this involves looking at whether you have a decent card (most if_em cards are decent), whether it's 32-bit or 64-bit PCI, and so on. For unidirectional send on 32-bit PCI, be aware that it is not possible to achieve gigabit performance because the PCI bus isn't fast enough, for example.

(3) Next, I'd use a tool like netperf (see ports collection) to establish three characteristics: round trip latency from user space to user space (UDP_RR), TCP throughput (TCP_STREAM), and large packet throughput (UDP_STREAM). With decent boxes on 5.3, you should have no trouble at all maxing out a single gig-e with if_em, assuming all is working well hardware wise and there's no software problem specific to your configuration.

(4) Note that router latency (and even switch latency) can have a substantial impact on gigabit performance, even with no packet loss, in part due to stuff like ethernet flow control. You may want to put the two boxes back-to-back for testing purposes.

(5) Next, I'd measure CPU consumption on the end box -- in particular, use top -S and systat -vmstat 1 to compare the idle condition of the system and the system under load.

If you determine there is a link layer or IP layer problem, we can start digging into things like the error statistics in the card, negotiation issues, etc. If not, you want to move up the stack to try and characterize where it is you're hitting the performance issue."

Richard Blum's book Network Performance Open Source Toolkit, which I read and reviewed last year, also gives good tips.

Snort 2.3.0 RC1 Released

Jeremy Hewlett announced the release of Snort 2.3.0 RC1. The major additions are the snort_inline code and the new sfportscan portscan detector.

Sguil users should recognize that alerts from the new portscan detector are not yet fully integrated, due to lack of support in Barnyard. Bamm is working on a modified op_sguil Barnyard component to support sfportscan output. If you enable sfportscan with Sguil, you'll see the alerts appear in the Sguil interface. However, they will not be inserted into the alerts database. This means sfportscan alerts will not be available for review once they are cleared from the display. Below is an example of how the new sfportscan alerts appear in Sguil.



The interesting aspect of the new sfportscan system is the creation of a "pseudo-packet" with the alert details. The alert above shows sfportscan believes ports 21 through 80 were scanned, although the activity which generated this report was directed at ports 21 through 25:

nmap -v -p 21-25 172.27.20.1-5

A look at the first Open Port alert shows sfportscan found port 21 open:



A look at the other Open Port alerts accurately reflects what the scan results showed.

Wednesday, November 17, 2004

Using x86info to Learn About HTT

Back in July I described my experiences running a June snapshot of FreeBSD 5-CURRENT on a Dell Poweredge 750 1U server. Recently I installed FreeBSD 5.3 RELEASE on the same hardware; here is the dmesg output for a kernel recompiled with SMP support.

While perusing the freebsd-current mailing list I came upon this thread which in part debated the merits of recompiling for SMP support on HTT machines. According to Intel:

"Hyper-Threading Technology, available on Intel Pentium 4 processors supporting Hyper-Threading Technology, is a form of simultaneous multithreading (SMT) that makes a single processor look like multiple processors to the operating system.

In its current implementation, HT Technology delivers two logical processors that can execute different tasks simultaneously using shared hardware resources. While an HT Technology–enabled processor doesn't equal the computing power of two physical processors, performance can be boosted with very little increase in cost and power consumption. For certain computing workloads, HT Technology improves performance with a much lower overhead of cost, power consumption, and space than the overhead required by multiple physical processors."

The poster in the freebsd-current thread needed a way to see how FreeBSD saw his HTT system. In other words, did it have "one" CPU or "two"? One reply suggested trying x86info, found in the ports tree.

Here is an example of how a true single CPU system appears to x86info:

bourque:/root# x86info
x86info v1.12b. Dave Jones 2001-2003
Feedback to .

Found 1 CPU
--------------------------------------------------------------------------
Family: 6 Model: 6 Stepping: 5 Type: 0 Brand: 0
CPU Model: Celeron (Mendocino) Original OEM
Instruction TLB: 4KB pages, 4-way associative, 32 entries
Instruction TLB: 4MB pages, fully associative, 2 entries
Data TLB: 4KB pages, 4-way associative, 64 entries
L2 unified cache:
Size: 128KB 4-way associative.
line size=32 bytes.
L1 Instruction cache:
Size: 16KB 4-way associative.
line size=32 bytes.
Data TLB: 4MB pages, 4-way associative, 8 entries
L1 Data cache:
Size: 16KB 4-way associative.
line size=32 bytes.

Here's how a true dual CPU system appears:

janney:/root# x86info
x86info v1.12b. Dave Jones 2001-2003
Feedback to .

Found 2 CPUs
--------------------------------------------------------------------------
CPU #1
/dev/cpu/0/cpuid: No such file or directory
Family: 6 Model: 7 Stepping: 3 Type: 0 Brand: 0
CPU Model: Pentium III (Katmai) [kC0] Original OEM
Instruction TLB: 4KB pages, 4-way associative, 32 entries
Instruction TLB: 4MB pages, fully associative, 2 entries
Data TLB: 4KB pages, 4-way associative, 64 entries
L2 unified cache:
Size: 512KB 4-way associative.
line size=32 bytes.
L1 Instruction cache:
Size: 16KB 4-way associative.
line size=32 bytes.
Data TLB: 4MB pages, 4-way associative, 8 entries
L1 Data cache:
Size: 16KB 4-way associative.
line size=32 bytes.
--------------------------------------------------------------------------
CPU #2
Family: 6 Model: 7 Stepping: 3 Type: 0 Brand: 0
CPU Model: Pentium III (Katmai) [kC0] Original OEM
Instruction TLB: 4KB pages, 4-way associative, 32 entries
Instruction TLB: 4MB pages, fully associative, 2 entries
Data TLB: 4KB pages, 4-way associative, 64 entries
L2 unified cache:
Size: 512KB 4-way associative.
line size=32 bytes.
L1 Instruction cache:
Size: 16KB 4-way associative.
line size=32 bytes.
Data TLB: 4MB pages, 4-way associative, 8 entries
L1 Data cache:
Size: 16KB 4-way associative.
line size=32 bytes.
--------------------------------------------------------------------------
WARNING: Detected SMP, but unable to access cpuid driver.
Used Uniprocessor CPU routines. Results inaccurate.

Here's how an HTT-enabled, single CPU system with a kernel recompiled for SMP appears:

fedorov:/root# x86info
x86info v1.12b. Dave Jones 2001-2003
Feedback to .

Found 2 CPUs
--------------------------------------------------------------------------
CPU #1
/dev/cpu/0/cpuid: No such file or directory
unknown TLB/cache descriptor: 0x60
Family: 15 Model: 3 Stepping: 4 Type: 0 Brand: 0
CPU Model: Unknown CPU Original OEM
Processor name string: Intel(R) Pentium(R) 4 CPU 2.80GHz

Instruction TLB: 4K, 2MB or 4MB pages, fully associative, 64 entries.
Data TLB: 4KB or 4MB pages, fully associative, 64 entries.
unknown TLB/cache descriptor: 0x60
No level 2 cache or no level 3 cache if valid 2nd level cache.
Instruction trace cache:
Size: 12K uOps 8-way associative.
L2 unified cache:
Size: 1MB Sectored, 8-way associative.
line size=64 bytes.
Processor serial: 0000-0F34-0000-0000-0000-0000
Number of logical processors supported within the physical package: 0

--------------------------------------------------------------------------
CPU #2
unknown TLB/cache descriptor: 0x60
Family: 15 Model: 3 Stepping: 4 Type: 0 Brand: 0
CPU Model: Unknown CPU Original OEM
Processor name string: Intel(R) Pentium(R) 4 CPU 2.80GHz

Instruction TLB: 4K, 2MB or 4MB pages, fully associative, 64 entries.
Data TLB: 4KB or 4MB pages, fully associative, 64 entries.
unknown TLB/cache descriptor: 0x60
No level 2 cache or no level 3 cache if valid 2nd level cache.
Instruction trace cache:
Size: 12K uOps 8-way associative.
L2 unified cache:
Size: 1MB Sectored, 8-way associative.
line size=64 bytes.
Processor serial: 0000-0F34-0000-0000-0000-0000
Number of logical processors supported within the physical package: 0

--------------------------------------------------------------------------
WARNING: Detected SMP, but unable to access cpuid driver.
Used Uniprocessor CPU routines. Results inaccurate.

Here's how the same hardware, with the default GENERIC kernel and no SMP, appears:

forsberg# x86info
x86info v1.12b. Dave Jones 2001-2003
Feedback to .

Found 1 CPU
--------------------------------------------------------------------------
unknown TLB/cache descriptor: 0x60
Family: 15 Model: 3 Stepping: 4 Type: 0 Brand: 0
CPU Model: Unknown CPU Original OEM
Processor name string: Intel(R) Pentium(R) 4 CPU 2.80GHz

Instruction TLB: 4K, 2MB or 4MB pages, fully associative, 64 entries.
Data TLB: 4KB or 4MB pages, fully associative, 64 entries.
unknown TLB/cache descriptor: 0x60
No level 2 cache or no level 3 cache if valid 2nd level cache.
Instruction trace cache:
Size: 12K uOps 8-way associative.
L2 unified cache:
Size: 1MB Sectored, 8-way associative.
line size=64 bytes.
Processor serial: 0000-0F34-0000-0000-0000-0000
Number of logical processors supported within the physical package: 0

There are two simple ways to know if your FreeBSD system is SMP-enabled. First, check top output. A single CPU box has no "C" column:

last pid: 30148; load averages: 0.00, 0.02, 0.00 up 9+04:05:35 22:05:56
22 processes: 1 running, 21 sleeping
CPU states: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle
Mem: 6304K Active, 127M Inact, 63M Wired, 60M Buf, 298M Free
Swap: 1024M Total, 1024M Free

PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
405 root 96 0 3440K 2792K select 0:05 0.00% 0.00% sendmail
421 root 8 0 1356K 1044K nanslp 0:01 0.00% 0.00% cron
288 root 96 0 1312K 904K select 0:01 0.00% 0.00% syslogd

Compare that listing with the following on a HTT box with SMP enabled. At the time top was running, sendmail and cron used "CPU one" and syslogd used "CPU zero":

last pid: 49096; load averages: 0.00, 0.00, 0.00 up 9+05:33:40 22:06:50
31 processes: 1 running, 30 sleeping
CPU states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
Mem: 10M Active, 152M Inact, 71M Wired, 16K Cache, 60M Buf, 261M Free
Swap: 1024M Total, 1024M Free

PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU CPU COMMAND
410 root 96 0 3460K 2800K select 1 0:08 0.00% 0.00% sendmail
426 root 8 0 1356K 1044K nanslp 1 0:02 0.00% 0.00% cron
291 root 96 0 1312K 868K select 0 0:01 0.00% 0.00% syslogd

On that same HTT SMP system, the dmesg output shows a "second CPU" launching:

SMP: AP CPU #1 Launched!

I intend to run my HTT systems with SMP enabled to see how they perform.

Saturday, November 13, 2004

Robert Watson Comments on FreeBSD 5.3 Networking Performance

While perusing the freebsd-questions mailing list, I read a post by Robert Watson describing FreeBSD 5.3's networking performance. In short, it seems FreeBSD 5.3 will process packets slower than FreeBSD 4.x, due to the transition to the SMPng architecture. Performance will improve over the next few months as optimizations are done within the new architecture. Tracking 5-STABLE will cause these improvements to be integrated into running systems.

I am deploying FreeBSD 5.3-based systems now, as the 5.x tree supports the Dell hardware I'm using. I have not had any problems with 5.x yet and I expect it to only get better.

Robert also mentioned a new FreeBSD Network Performance site, describing in detail the work done to enhance the performance of the FreeBSD network stack.

Sun Java Desktop System Live CD

I downloaded the .iso for the live CD version of the Sun Java Desktop System. When booting it reports being based on the Linux distro Morphix, version 0.4.

I was sad to see the X autoconfiguration fail on my Thinkpad a20p, so I couldn't test it on my primary system. I was able to boot the CD on my Shuttle SB52G2 and on a Dell Optiplex GX100. On the Shuttle JDS was not able to automatically detect my Linksys WMP55AG A+G Wireless PCI Adapter. I didn't feel like fiddling around with Linux so I wasn't able to access the network on the Shuttle.

Since the Dell had a recognized Ethernet NIC, I was able to manually configure it for network access. JDS (like most live CDs) tries to find an IP via DHCP, but I don't provide DHCP on my test LAN. Launching a terminal showed me I was running as user 'morph', which means Sun didn't change even the basics of this Morphix version. I was able to 'su -' without a password and configure the network myself.

If your hardware is supported, I think JDS is a fairly slick distro for the average office worker. StarOffice 7 is included. Here are screenshots. Sun is throwing lots of energy at this effort, with support forums, a blog, and an O'Reilly book.

Friday, November 12, 2004

FreeBSD for Linux Users

Dru Lavigne published an excellent article called FreeBSD for Linux Users. She does a nice job explaining run levels, kernel differences, startup scripts, package installation, and documentation. She plans to follow up the article with one on similarities between FreeBSD and Linux.

Thursday, November 11, 2004

External 3.5 Hard Drive Enclosure

I previously wrote about my experiences using an external 2.5 HDD enclosure. Today I received my AMS Electronics Venus DS3 external 3.5 HDD enclosure. I liked this model because it supports Firewire and USB 2.0, has an on-off switch, and provides an internal fan to cool hotter drives. I found mounting the drive inside the enclosure very easy and the product itself seems sturdy. The on-off switch should have been a rocker switch and not a somewhat weaker toggle, but it works ok.

On my laptop, when attaching the DS3 to the Firewire port on my Adaptec Duo Connect Firewire/USB adapter, dmesg reports the following:

fwohci0: BUS reset
fwohci0: Async DMA Receive error err = 1f
fwohci0: node_id=0xc000ffc1, gen=3, CYCLEMASTER mode
firewire0: 2 nodes, maxhop <= 1, cable IRM = 1 (me)
firewire0: bus manager 1 (me)
firewire0: New S400 device ID:0030e0f4e0204651
da0 at sbp0 bus 0 target 0 lun 0
da0: Fixed Simplified Direct Access SCSI-4 device
da0: 50.000MB/s transfers
da0: 4110MB (8418816 512 byte sectors: 255H 63S/T 524C)
(da0:sbp0:0:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da0:sbp0:0:0:0): ABORTED COMMAND csi:ff,f,0,e0 asc:20,0
(da0:sbp0:0:0:0): Invalid command operation code
(da0:sbp0:0:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da0:sbp0:0:0:0): ABORTED COMMAND csi:43,5f,5f,54 asc:20,0
(da0:sbp0:0:0:0): Invalid command operation code
(da0:sbp0:0:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da0:sbp0:0:0:0): ABORTED COMMAND csi:ff,f,0,b0 asc:20,0
(da0:sbp0:0:0:0): Invalid command operation code
(da0:sbp0:0:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da0:sbp0:0:0:0): ABORTED COMMAND csi:ff,f,0,40 asc:20,0
(da0:sbp0:0:0:0): Invalid command operation code
(da0:sbp0:0:0:0): Invalid command operation code
(da0:sbp0:0:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da0:sbp0:0:0:0): ABORTED COMMAND csi:ff,f,0,70 asc:20,0
(da0:sbp0:0:0:0): Invalid command operation code
(da0:sbp0:0:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da0:sbp0:0:0:0): ABORTED COMMAND csi:ff,1f,0,70 asc:20,0
(da0:sbp0:0:0:0): Invalid command operation code
(da0:sbp0:0:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da0:sbp0:0:0:0): ABORTED COMMAND csi:ff,f,0,40 asc:20,0
(da0:sbp0:0:0:0): Invalid command operation code

I was successfully able to mount the hard drive and then copy large (~60 MB) files to and from the drive, verifying their integrity using md5. However, the kernel reported these errors:

sbp0:0:0 No ocb(cca634) on the queue
sbp0:0:0 No ocb(cca9dc) on the queue
sbp0:0:0 No ocb(cca01c) on the queue
sbp0:0:0 No ocb(cca154) on the queue
sbp0:0:0 No ocb(cca154) on the queue
sbp0:0:0 No ocb(cca154) on the queue

I tried disabling Tagged Queueing but the errors seemed to continue.

I found that attaching the device to the USB 1.1 port on the laptop itself worked, albeit with much slower transfer speeds. Here's how the kernel saw the device:

da0 at umass-sim0 bus 0 target 0 lun 0
da0: Fixed Direct Access SCSI-0 device
da0: 1.000MB/s transfers
da0: 4110MB (8418816 512 byte sectors: 255H 63S/T 524C)

I had better results attaching the external HDD enclosure via Firewire to my Dell 2300 server via Adaptec Duo Connect PCI card. I saw the same error messages beginning with SYNCHRONIZE CACHE but none of the 'queue' errors.

Now that FreeBSD has started the freebsd-usb mailing list, I believe USB support will improve. I've never had much luck with USB 2 support using the ehci driver, which isn't in the default GENERIC kernel and must be added in via recompiling the kernel.