FreeBSD Networking over FireWire

You might be familiar with Apple's implementation of IP over FireWire. This allows connecting two computers directly over FireWire ports. FreeBSD offers two drivers that provide networking over FireWire. fwe is a non-standard protocol, but it is implemented by default in the GENERIC kernel. fwip implements RFC 2734 (IPv4 over IEEE 1394) and RFC 3146 (Transmission of IPv6 Packets over IEEE 1394 Networks); it is available via kernel module.

I decided to have my laptop orr talk to my server janney using FireWire. To implement FireWire, orr uses an Adaptec DuoConnect PC Card Adapter and janney uses an Adaptec DuoConnect PCI Adapter. Both provide FireWire and USB 2.0.

Each system is running FreeBSD 6.0.

The laptop dmesg sees the following when the FireWire adapter is inserted.

cardbus0: CIS pointer is 0!
cardbus0: Resource not specified in CIS: id=10, size=800
cardbus0: Resource not specified in CIS: id=14, size=4000
fwohci0:  mem 0x88004000-0x880047ff,0x88000000-0x88
003fff irq 11 at device 0.0 on cardbus0
fwohci0: OHCI version 1.10 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 08:00:28:56:02:00:49:8a
fwohci0: Phy 1394a available S400, 3 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0:  on fwohci0
fwe0:  on firewire0
if_fwe0: Fake Ethernet address: 0a:00:28:00:49:8a
fwe0: Ethernet address: 0a:00:28:00:49:8a
fwe0: if_start running deferred for Giant
sbp0:  on firewire0
fwohci0: Initiate bus reset
fwohci0: node_id=0xc000ffc0, gen=1, CYCLEMASTER mode
firewire0: 1 nodes, maxhop <= 0, cable IRM = 0 (me)
firewire0: bus manager 0 (me)
cardbus0: CIS pointer is 0!
cardbus0: Resource not specified in CIS: id=10, size=1000
ohci0:  mem 0x88000000-0x88000fff irq 11 at device
0.4 on cardbus0
ohci0: [GIANT-LOCKED]
usb1: OHCI version 0.0
usb1: unsupported OHCI revision
ohci0: USB init failed

The server dmesg sees the following at boot.

ohci0:  mem 0xf7fff000-0xf7ffffff irq 22 at device 
8.0 on pci4
ohci0: [GIANT-LOCKED]
usb1: OHCI version 1.0
usb1:  on ohci0
usb1: USB revision 1.0
uhub1: NEC OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 3 ports with 3 removable, self powered
ohci1:  mem 0xf7ffe000-0xf7ffefff irq 21 at device 
8.1 on pci4
ohci1: [GIANT-LOCKED]
usb2: OHCI version 1.0
usb2:  on ohci1
usb2: USB revision 1.0
uhub2: NEC OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
ehci0:  mem 0xf7ffdc00-0xf7ffdcff irq 20 at d
evice 8.2 on pci4
ehci0: [GIANT-LOCKED]
usb3: EHCI version 0.95
usb3: companion controllers, 3 ports each: usb1 usb2
usb3:  on ehci0
usb3: USB revision 2.0
uhub3: NEC EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub3: 5 ports with 5 removable, self powered
fwohci0:  mem 0xf7ffd000-0xf7ffd7ff,0xf7ff8000-0xf7
ffbfff irq 22 at device 12.0 on pci4
fwohci0: OHCI version 1.10 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:50:42:b5:c0:0d:0d:af
fwohci0: Phy 1394a available S400, 3 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0:  on fwohci0
fwe0:  on firewire0
if_fwe0: Fake Ethernet address: 02:50:42:0d:0d:af
fwe0: Ethernet address: 02:50:42:0d:0d:af
fwe0: if_start running deferred for Giant
sbp0:  on firewire0
fwohci0: Initiate bus reset
fwohci0: node_id=0xc800ffc1, gen=1, CYCLEMASTER mode
firewire0: 2 nodes, maxhop <= 1, cable IRM = 1 (me)
firewire0: bus manager 1 (me)

Here is the fwe interface that is created when the FireWire adapter is active on orr.

fwe0: flags=108802 mtu 1500
        options=8
        ether 0a:00:28:00:49:8a
        ch 1 dma -1

Here is the fwe interfaced that is created when the FireWire adapter is active on janney.

fwe0: flags=108943 mtu 1500
        options=8
        inet6 fe80::50:42ff:fe0d:daf%fwe0 prefixlen 64 scopeid 0x2 
        inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255
        ether 02:50:42:0d:0d:af
        ch 1 dma 0

When I plug a FireWire cable into each host, dmesg on orr sees the following.

fwohci0: BUS reset
fwohci0: node_id=0xc800ffc1, gen=2, CYCLEMASTER mode
firewire0: 2 nodes, maxhop <= 1, cable IRM = 1 (me)
firewire0: bus manager 1 (me)
firewire0: New S400 device ID:005042b5c00d0daf

Server janney sees similar messages.

fwohci0: BUS reset
fwohci0: node_id=0x8800ffc0, gen=4, non CYCLEMASTER mode
firewire0: 2 nodes, maxhop <= 1, cable IRM = 1
firewire0: bus manager 1 
firewire0: New S400 device ID:080028560200498a

First I assign an IP to fwe0 on orr.

orr:/home/richard$ sudo ifconfig fwe0 inet 10.1.1.1 netmask 255.255.255.0 up
orr:/home/richard$ ifconfig fwe0
fwe0: flags=108943 mtu 1500
        options=8
        inet6 fe80::800:28ff:fe00:498a%fwe0 prefixlen 64 scopeid 0x4 
        inet 10.1.1.1 netmask 0xffffff00 broadcast 10.1.1.255
        ether 0a:00:28:00:49:8a
        ch 1 dma 0

Next I assign an IP to fwe0 on janney.

janney:/home/richard$ sudo ifconfig fwe0 inet 10.1.1.2 netmask 255.255.255.0 up
janney:/home/richard$ ifconfig fwe0
fwe0: flags=108943 mtu 1500
        options=8
        inet6 fe80::50:42ff:fe0d:daf%fwe0 prefixlen 64 scopeid 0x2 
        inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255
        ether 02:50:42:0d:0d:af
        ch 1 dma 0

Now the two systems can communicate over FireWire.

janney:/home/richard$ ping 10.1.1.1
PING 10.1.1.1 (10.1.1.1): 56 data bytes
64 bytes from 10.1.1.1: icmp_seq=0 ttl=64 time=0.694 ms
64 bytes from 10.1.1.1: icmp_seq=1 ttl=64 time=0.333 ms
64 bytes from 10.1.1.1: icmp_seq=2 ttl=64 time=0.312 ms
^C
--- 10.1.1.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.312/0.446/0.694/0.175 ms

One can sniff the fwe0 interface with Tcpdump.

07:36:42.479420 IP6 fe80::50:42ff:fe0d:daf > ff02::1:ff0d:daf: HBH ICMP6, multicast listener 
 reportmax resp delay: 0 addr: ff02::1:ff0d:daf, length 24
07:36:42.479667 IP6 fe80::50:42ff:fe0d:daf > ff02::2:f2f0:bec6: HBH ICMP6, multicast listener
 reportmax resp delay: 0 addr: ff02::2:f2f0:bec6, length 24
07:36:42.479818 arp who-has 10.1.1.2 tell 10.1.1.2
07:36:42.498496 IP6 :: > ff02::1:ff0d:daf: ICMP6, neighbor solicitation, who has
 fe80::50:42ff:fe0d:daf, length 24
07:36:43.756817 IP6 fe80::50:42ff:fe0d:daf > ff02::1:ff0d:daf: HBH ICMP6, multicast listener 
 reportmax resp delay: 0 addr: ff02::1:ff0d:daf, length 24
07:36:50.962703 IP6 fe80::50:42ff:fe0d:daf > ff02::2:f2f0:bec6: HBH ICMP6, multicast listener
 reportmax resp delay: 0 addr: ff02::2:f2f0:bec6, length 24
07:36:54.422718 arp who-has 10.1.1.1 tell 10.1.1.2
07:36:54.422793 arp reply 10.1.1.1 is-at 0a:00:28:00:49:8a
07:36:54.423012 IP 10.1.1.2 > 10.1.1.1: ICMP echo request, id 34306, seq 0, length 64
07:36:54.423050 IP 10.1.1.1 > 10.1.1.2: ICMP echo reply, id 34306, seq 0, length 64

You can see that the fwe driver supports IPv6. The last four packets are IPv4.

Next I decided to try the fwip driver. I loaded the fwip kernel module on each system. First, orr.

orr:/home/richard$ sudo kldload fwip
orr:/home/richard$ kldstat
Id Refs Address    Size     Name
 1   12 0xc0400000 63072c   kernel
 2    2 0xc0a31000 74b0     snd_csa.ko
 3    3 0xc0a39000 1d408    sound.ko
 4    1 0xc0a57000 c3a4     r128.ko
 5    2 0xc0a64000 eeec     drm.ko
 6   16 0xc0a73000 568dc    acpi.ko
 7    1 0xc2236000 5000     if_fwip.ko

This created the fwip0 interface, which I gave an IP address.

orr:/home/richard$ ifconfig fwip0
fwip0: flags=108802 mtu 1500
        lladdr 8.0.28.56.2.0.49.8a.a.2.ff.fe.0.0.0.0
orr:/home/richard$ sudo ifconfig fwip0 inet 172.16.1.1 netmask 255.255.255.0 up
orr:/home/richard$ ifconfig fwip0
fwip0: flags=108843 mtu 1500
        inet6 fe80::a00:2856:200:498a%fwip0 prefixlen 64 scopeid 0x5 
        inet 172.16.1.1 netmask 0xffffff00 broadcast 172.16.1.255
        lladdr 8.0.28.56.2.0.49.8a.a.2.ff.fe.0.0.0.0

Notice dmesg output for orr.

fwip0:  on firewire0
fwip0: Firewire address: 08:00:28:56:02:00:49:8a @ 0xfffe00000000, S400, maxrec 2048

Now, janney.

janney:/home/richard$ sudo kldload fwip
janney:/home/richard$ kldstat 
Id Refs Address    Size     Name
 1    5 0xc0400000 641298   kernel
 2   16 0xc0a42000 568dc    acpi.ko
 3    1 0xc214a000 5000     if_fwip.ko
janney:/home/richard$ ifconfig fwip0
fwip0: flags=108802 mtu 1500
        lladdr 0.50.42.b5.c0.d.d.af.a.2.ff.fe.0.0.0.0
janney:/home/richard$ sudo ifconfig fwip0 inet 172.16.1.2 netmask 255.255.255.0 up
janney:/home/richard$ ifconfig fwip0
fwip0: flags=108843 mtu 1500
        inet6 fe80::250:42b5:c00d:daf%fwip0 prefixlen 64 scopeid 0x6 
        inet 172.16.1.2 netmask 0xffffff00 broadcast 172.16.1.255
        lladdr 0.50.42.b5.c0.d.d.af.a.2.ff.fe.0.0.0.0

Notice janney's dmesg.

fwip0:  on firewire0
fwip0: Firewire address: 00:50:42:b5:c0:0d:0d:af @ 0xfffe00000000, S400, maxrec 2048

Now the two systems can communicate using the fwip driver.

janney:/home/richard$ ping -c 1 172.16.1.1
PING 172.16.1.1 (172.16.1.1): 56 data bytes
64 bytes from 172.16.1.1: icmp_seq=0 ttl=64 time=0.733 ms

--- 172.16.1.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.733/0.733/0.733/0.000 ms

You can also sniff fwip0. Take a look at the differences.

orr:/home/richard$ tcpdump -n -r fwe0.short.lpc -c 2 icmp
reading from file fwe0.short.lpc, link-type EN10MB (Ethernet)
07:36:54.423012 IP 10.1.1.2 > 10.1.1.1: ICMP echo request, id 34306, seq 0, length 64
07:36:54.423050 IP 10.1.1.1 > 10.1.1.2: ICMP echo reply, id 34306, seq 0, length 64

orr:/home/richard$ tcpdump -n -r fwip0.short.lpc -c 2 icmp
reading from file fwip0.short.lpc, link-type APPLE_IP_OVER_IEEE1394 (Apple IP-over-IEEE 1394)
07:41:52.593940 IP 172.16.1.2 > 172.16.1.1: ICMP echo request, id 38914, seq 0, length 64
07:41:52.593970 IP 172.16.1.1 > 172.16.1.2: ICMP echo reply, id 38914, seq 0, length 64

The first reports "Ethernet" because fwe is an Ethernet emulation layer. The second reports Apple IP-over-IEEE 1394 because that is an entirely new protocol. Check it out in Tethereal.

Frame 8 (102 bytes on wire, 102 bytes captured)
    Arrival Time: Jan 30, 2006 07:41:52.593940000
    Time delta from previous packet: 0.000231000 seconds
    Time since reference or first frame: 13.327087000 seconds
    Frame Number: 8
    Packet Length: 102 bytes
    Capture Length: 102 bytes
    Protocols in frame: ap1394:ip:icmp:data
Apple IP-over-IEEE 1394, Src: 005042B5C00D0DAF, Dst: 80B2FFC100000000
    Destination: 80B2FFC100000000
    Source: 005042B5C00D0DAF
    Type: IP (0x0800)
Internet Protocol, Src: 172.16.1.2 (172.16.1.2), Dst: 172.16.1.1 (172.16.1.1)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 84
    Identification: 0x0696 (1686)
    Flags: 0x00
        0... = Reserved bit: Not set
        .0.. = Don't fragment: Not set
        ..0. = More fragments: Not set
    Fragment offset: 0
    Time to live: 64
    Protocol: ICMP (0x01)
    Header checksum: 0x19f0 [correct]
        Good: True
        Bad : False
    Source: 172.16.1.2 (172.16.1.2)
    Destination: 172.16.1.1 (172.16.1.1)
Internet Control Message Protocol
    Type: 8 (Echo (ping) request)
    Code: 0 
    Checksum: 0xe88d [correct]
    Identifier: 0x9802
    Sequence number: 0x0000
    Data (56 bytes)

0000  43 de 09 a3 00 09 3e e2 08 09 0a 0b 0c 0d 0e 0f   C.....>.........
0010  10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f   ................
0020  20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f    !"#$%&'()*+,-./
0030  30 31 32 33 34 35 36 37                           01234567

I noticed each host reported error messages like the following. I believe these were caused by the fwip driver.

fwohci0: txd err= 3 miss Ack err

I believe fwip has a future going forward, since it implements a standard.

For a rough idea of how fast these interfaces were, I transferred the same 100 MB file using the fwe, fwip, and fxp/xl wired Ethernet interfaces. Here are the results.

1. fwe: 104857600 bytes received in 00:09 (10.16 MB/s)


2. fwip: 104857600 bytes received in 00:09 (11.01 MB/s)


3. fxp/xl through a switch: 104857600 bytes received in 00:08 (11.21 MB/s)


4. fxp/xl via crossover cable: 104857600 bytes received in 00:09 (10.75 MB/s)

All the results are roughly the same, so the bottleneck is probably the hard drive of the laptop.

This opens some interesting possibilities for networking. Maybe I will buy a FireWire hub and connect multiple machines simultaneously.

I hope to try some of the debugging and memory access features of FireWire in the future.