About AoE

ATA over Ethernet or AoE is relatively new protocol, witch is descriped as “puts ATA disk commands directly into Ethernet frames”. Ethernet frames are non-routable, which provides inherent security (According to Coraid Inc.).

What it basically does is putting block devices directly on the network, without the TCP overhead, making the remote devices accesable as local devices. Coraid Inc. makes hardware devices called “Coraid EtherDrive Storage Blades” but they have been kind enough to
release their software under the GPL, including drivers for Linux (included in the standard kernel tree from version 2.6.11) and FreeBSD, userspace tools and not least a software only server, enabling you to export whatever block devices you have via AoE.

Install the software

I have created ebuilds to do the installation in Gentoo Linux. The overlay is currently available as http://hilli.dk/files/aoe-overlay.tar.gz. Download this, extract it to a suiting directory (i.e. /usr/local/portage/aoe-overlay) and add this directory to the PORTDIR_OVERLAY variable in /etc/make.conf indigenerics.com.

The server

Install sys-fs/vblade:

# ACCEPT_KEYWORDS=~arch emerge vblade

Replace “arch” with the relevant keyword for your architecture.

Starting the server is simple:

# /usr/sbin/vblade 1 0 eth0 /dev/hda2
ioctl returned 0
4194892800 bytes
pid 12014: e1.0, 8193150 sectors

What happens here is that you start the server as shelf 1, slot is 0, the ethernet device used is eth0 and the exported filesystem is /dev/hda2. You pick the shelf and slot number yourself. I suggest using the shelf number pr server and a slot number per block device you are exporting. If your setup is going to be large enough, I also suggest using a seperate network (And a fast one too).

The client

It is enough to compile the kernel (>=2.6.11) with the ATA_OVER_ETH option as either built-in or as a module. Pick the module option for now. You will find it under

Device Drivers
-> Block devices
-> ATA over Ethernet support

But if you install the aoetools package you will be a bit better off.

# ACCEPT_KEYWORDS=~arch emerge aoetools

Again replace “arch” with the relevant keyword for your architecture.

Insert your new aoe module, and watch for stuff in your syslog.

# modprobe aoe
# dmesg | tail -n 3
aoe: aoe_init: AoE v2.6-5 initialised.
etherd/e1.0:
aoe: 0080c8caff65 e1.0 v4000 has 8193150 sectors
# aoe-stat
e1.0 eth0 up

This shows a few things: The kernel have allready discovered the remotely exported device. And that you can use aoe-stat to display a list of available devices.
This new device is now know as /dev/etherd/e1.0.
Partition it and make a filesystem on it; Creating one partition as the first partition should yield /dev/etherd/e1.0p1:

# fdisk -l /dev/etherd/e1.0

Disk /dev/etherd/e1.0: 4194 MB, 4194892800 bytes
255 heads, 63 sectors/track, 510 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/etherd/e1.0p1 1 510 4096543+ 83 Linux

Format it:

# mkreiserfs /dev/etherd/e1.0p1
mkreiserfs 3.6.19 (2003 www.namesys.com)

A pair of credits:
Continuing core development of ReiserFS is mostly paid for by Hans Reiser from
money made selling licenses in addition to the GPL to companies who don’t want
it known that they use ReiserFS as a foundation for their proprietary product.
And my lawyer asked ‘People pay you money for this?’. Yup. Life is good. If you
buy ReiserFS, you can focus on your value add rather than reinventing an entire
FS.

Alexander Lyamin keeps our hardware running, and was very generous to our
project in many little ways.

Guessing about desired format.. Kernel 2.6.11-gentoo-r6 is running.
Format 3.6 with standard journal
Count of blocks on the device: 1024128
Number of blocks consumed by mkreiserfs formatting process: 8243
Blocksize: 4096
Hash function used to sort names: “r5”
Journal Size 8193 blocks (first block 18)
Journal Max transaction length 1024
inode generation number: 0
UUID: 66dd3154-349a-4a42-9b8a-b544cfef862f
ATTENTION: YOU SHOULD REBOOT AFTER FDISK!
ALL DATA WILL BE LOST ON ‘/dev/etherd/e1.0p1’!
Continue (y/n):y
Initializing journal – 0%….20%….40%….60%….80%….100%
Syncing..ok

Tell your friends to use a kernel based on 2.4.18 or later, and especially not a
kernel based on 2.4.9, when you use reiserFS. Have fun.

ReiserFS is successfully created on /dev/etherd/e1.0p1.

Security

To be done…