Hey guys,

I’m running my FreeBSD server now for around 1 month using ZFS for all my data. I was fascinated by ZFS for a long time (well, for a few days even), but with the time, it seems that there are 2 things that are not so well as expected using ZFS.

Painpoint 1 [Speed]
Yes, ZFS with an zpool using RAIDz1 (RAID5 similar) is a lot of work for the CPU and a lot thing to buffer for the RAM. It’s not that I wasn’t warned by the docs, 1GB of RAM is mandatory for running RAIDz1. The reason is quiet simple explained:

RAIDz1 needs to check itself, the so-called “self healing process”. That costs a lot of power and time. Reading and comparing the checksums (which is the magic behind the security) is time expensive. The result?

Well, writing files to my UFS partition (system disk) means to be able to put files with 80-90MByte/s through the GBit network onto this disk.

Same line, same source, but different disk(s) (now the zpool) ends up in putting just 30-40MByte/s through the network.

Both tests are done using an NFS share between my Mac Pro and the FreeBSD server in the basement.

Well, that’s odd and a bit frustrating for me, because it took me a few hours of work to extend the network from 100MBit/s Level to 1GBit/s Level by checking every single connection in the plugs and beeing sure everything is grounded.

Painpoint 2 [Expansion issues]
Sure, ZFS/RAIDz1 is no Drobo, so you can’t easily hot plug drives in and out and get “full” usage of the new disks by doing so. That’s ok, but the really annoying thing is, that you cannot expand your RAIDz1 zpool afterwards. If you want to expand your zpool, you’ll need to store your files somewhere (sure, it is recommended to have a current backup everytime, so that shouldn’t be the problem for the people who are doing this naturally) and destroy the old zpool and create the new one with the new disk.

That’s ok if you have for example 3x 1TB disks and want to add at least 1 more disk to have a final size of around about 2.8TB available and then you’re finished with that project. But if you want to storage to grow, there’s just one (expensive) chance to do this:

Buy at least 3 1TB disks and build a seconds zpool. Then you are able to mix both zpools to one large. That’s expensive, maybe a nice solution for a company, because that would result in something called a RAIDz2 / RAID6 similar with 2 parity disks. But for easy growing, ZFS is not the filesystem you’ve dreamed off. It’s easier to do those things with LVM on Linux for example. Haven’t used LVM on more than one disk for myself honestly, so I can’t say anything about security regarding LVM.

Painpoint 3 [FreeBSD]
FreeBSD is cool, isn’t it? It’s more or less to administrate and it’s rock solid. Yes it is, but the first thing, easy to administrate, I must say: no. Maybe because of my knowledge of Debian, but I really miss a simply “apt-get install apache2″ and a few more apt-get’s to install a great running webserver.

I’m not really sure and I think I will take this day off to think about it and what are the best practices. On the one hand, the ZFS is damn secure, on the other hand, having services running on FreeBSD is hard hard work for me.

Well – enjoy your sunday, I’ll do and I’m happy to have the upcoming week free for holidays This means: sleeping long and have more time to rethink this all (or simply switching off the computers and having a great time with my wife ).

Ciao
Dennis

PS. I just got a very rude comment on Reddit about this article. I want to make clear, that I don’t blame the software (FreeBSD nor ZFS) in ANY way! It’s great software, but this article shows my current experiences with it. I just wanted to make this clear!

Tweet This Post Digg This Facebook Reddit

Good morning guys,

Saesh has asked me, if I’ve tried using NFS from FreeBSD and OS X as client. Well, I had it in mind, but haven’t tested it yet. So, yesterday morning I gave it a try.

NFS comes integrated in FreeBSD 7.1 (which is the system that I use on the fileserver). All you have to do to start it up are those steps:

Add those 3 lines to your /etc/rc.conf
rpcbind_enable="YES"
nfs_server_enable="YES"
mountd_flags="-r"

Next, create your own /etc/exports in the format:
/storage 192.168.0.123

This will allow the client with the IP 192.168.0.123 to access the fileserver’s share (path: /storage) via NFS. Also in Leopard, NFS is built it, but there are a few more steps to do before you can try to log on.

Most important: Check if nfsd (NFS daemon) is running yet, if so – kill the processes.

# ps axu | grep nfsd
root 8918 0.0 0.0 4604 1380 ?? Is 2:30PM 0:00.03 nfsd: master
root 8919 0.0 0.0 4604 1072 ?? I 2:30PM 0:35.74 nfsd: server
root 8920 0.0 0.0 4604 1072 ?? I 2:30PM 0:18.33 nfsd: server
root 8921 0.0 0.0 4604 1072 ?? I 2:30PM 0:16.56 nfsd: server
root 8922 0.0 0.0 4604 1072 ?? I 2:30PM 0:16.20 nfsd: server
root 25658 0.0 0.0 5928 1220 p0 R+ 8:05AM 0:00.00 grep nfsd
# kill -9 8918 8919 8920 8921 8922

Ok – now you’re ready for the final initialization

# /etc/rc.d/mountd onereload
# rpcbind
# nfsd -u -t -n 4
# mountd -r

That’s it. Check your sharepoints with a simple
# showmount -e
Exports list on localhost:
/storage 192.168.0.123

You should be now able to mount your NFS share from OS X Leopard. But now, my problems begin. I’m currently now able to read/write from Leopard. Why? Well, I’m sure it’s again the every-time-killing-my-nerves user rights that annoys me here again.

But, slowly. On the Mac, press CMD+K, when you’ve selected the Finder. A popup will appear.

Mounting this should work and you will possibly get the following message, which means, that you don’t have rights to access this folder (*sigh* )

At the finder, it looks like this (do you see the little red sign inside the icon?).

What I’ve done is to create a new “share” or “export” in the /etc/exports of my Backup drive (2x 200GB which is more a testdrive with ZFS RAIDz1, too). I set the complete Volume to

# chmod 777 -R /backup

Then I mount it (don’t forget to)
# /etc/rc.d/mountd onereload
before you mount! Otherwise you will not be able to mount

Looks like it works, ha? Yes – a bit. I can now access the Backup Volume and copy a file from this share to my harddisk to test the speed. I’ve fixed the connection between the home office (attic) and the basement (server rack), so full 1GBit/s is available. I was able to copy the files with a rate of 70-80MByte/s from the server to the local disk inside my Mac.

That’s damn fast and a lot lot faster than SMB.

The bad thing is, I’m not able to write files onto this disk. Yes – since “chmod 777″ allows me, I’m able to create folders and also copy files to the disc, but I’m not able to use them. Let me show you an example:

I drag a file from my desktop to this share (fully “777″!)

You possibly need to identify yourself as root… (short version)

The file that you want to copy contains at least one object wherefor you have just the rights to read. Do you want to copy that file?

Well, finally it’s placed and looks good…

…but to open it, won’t work “The file you are trying to open is damaged or the format is unknown”

Uh! Nice message for a simple PNG file, isn’t it? Well, that’s the point where I am at the moment. The NFS server is running, but I can’t really access my files. Sure, I can open the already stored files without problems, but adding new? Currently not possible.

Oh well, I found examples like this for the /etc/exports (mostly for Linux):
/storage 192.168.0.0/24(rw,sync,insecure)

This allows everyone from the 192.168.0.x network to access those files with read/write permissions. Insecure is a command that should work with Leopard. Nice! I tried it, but when I type “showmount -e” afterwards (and after refreshing mountd), I see NO shares, also Leopard is no more able to mount anything.

Ideas and help is really appreciated

Ciao
Dennis

Tweet This Post Digg This Facebook Reddit

G’morning guys,

do you know the feeling when a system is so rock solid that you having nothing to complain about?

Well, me not – yet. But it seems, that FreeBSD is such a system. Let me explain what happend.

The installed OS (Mac OS X Leopard 10.5.6) on my Client/Workstation is basicly installed since August last year. Yep! I moved it using Carbon Copy Cloner over 3 harddisks so far, because I was in need of the disk space for the backup for the server preparation (to name it: all 3x 1TB disks were in use and no more available for backups). So – this morning something strange happend. I’m gonna use a Samba/SMB/CIFS currently for the main transfer of files and the videostuff between the Mac and the FreeBSD server. I was just in Photoshop to save a the photo of the X41 keyboard that I used for my this mornings update on the last post. Suddenly it told me that it can’t save. Error. Booom. I thought “Damn – what’s this?”, opened up a terminal and connected to the server using the GBit Subnet IP. Works fine, a Samba restart didn’t helped me out. I was about to think about a server reboot, but wait. Since 2 days, my QuickSilver didn’t index as regular, typing program names delivers anything else, but not what I was looking for. Finally, I shut the Mac down, give a few seconds to rest and launched it up. To explain: I’m using the energy save mode and rarly really shut down the Mac. So the uptime was about 2 weeks or so and the RAM was filled up to 10GB because of different very big tools for videocut.

Booting it up again didn’t showed me my Storage Samba icon. Oh dear! It’s a server problem. But, hold on – pressing Alt+K to connect to the server and voilá – it works. Normally it should mount itself on the systems start, but I think that I’ve used the wrong account credentials to login when I created this shortcut.

Icon back, everything fine and works.

Resumé: The meanwhile 5 days running server (yes 24/7), works still perfect. I think I should find a job for him, like doing SETI@home or Folding@home or Climate Prediction jobs using BOINC.

Desired icon of luck

In the last few days, I really thought about switching to OpenSolaris on the server because of the bit better implemenation of ZFS into the system, but – it’s rock solid, so why should I do if everything works fine? FreeBSD seems really to be one of those (todays rare) “Fire & Forgot” systems.

Just Pros for this OS? Hmmnaaaa…. not all is fine. It’s a server OS in my mind. I used to install it on my X41 as mentioned in my last post, it’s fine and after fiddling around for a few hours and the great help of some guys from the #bsdgroup.de channel on irc.freenode.org, the WiFi also worked flawless, but what’s this? pkg_add and using the ports just give me Firefox 2.0? Yes – Firefox is even on my Mac not the nice and stable sweet part of software that I know well from the past (it really tends to crash and I’m just using AdBlock Plus, Filter Updater, Foxmarks and Flash as plugins).

While I’m waiting for the recovery CDs to arrive, I think I will play with other systems, I’ve laying around here. Cryptic stuff like QNX or so Nothing to work with, but interessting to play with, isn’t it?

For your amusement, here’s, to return to the server site of this post, a few videos I found on YouTube about ZFS. I really enjoyed them and they finally helped me to make the decision to use ZFS and RAIDz1 for my data.

Have fun!

Ciao
Dennis

Tweet This Post Digg This Facebook Reddit

Hi guys,

so – this is new for this blog and also new for me Here’s my first videoblog for klein2. Enjoy it!

Ciao
Dennis

Tweet This Post Digg This Facebook Reddit

Good morning,

I’ve played with and tested different types of server OS in the last two weeks. You’ve noticed it; my very first installation (since switching away from Debian) was Windows Server 2008. It was nice and fast, but an evalution. Then we (yeah that’s me and my wife) had the idea that using an Exchange Server at home could be very nice. So I spend another few days with poking around with Windows Server 2003 SBS that we already had and next with Windows Server for Small Business 2008 (x64) as another evaluation copy. Not to mention that it was a horrible act to keep backup of our content. In the end, anything failed extremely and made a strike through the bill (not sure if I can really translate this saying into english, but in German it means, that the plan has failed).

The most important thing for the server should be file storage. Secure file storage! Well, using Windows Server allowed me to use the onboard RAID0/1 controller to create a mirror of 2x 1TB drives. Nice, but.. I wanted to go to RAID5 and there’s also the costs. Not just for a good RAID card which costs around 350 – 400 Euro (PCIexpress, 3ware), no it’s also the licence for a Windows Server which is pretty expensive. Too expensive for a server that serves just 7 computers in our and the neighbours house.

So, I turned to Google and typed “Good OS for a fileserver”, and guess what it said? FreeBSD. Sure, FreeBSD… I tried that years ago – as desktop and it was nice and easy (compared to Linux very easy and much better structured), but it was never an OS that I would recommend for a desktop. Yes – that depends on the kind of applications that I run: Photoshop, Illustrator and so on. Nothing to go on FreeBSD here (please, don’t start discussing about GIMP – I like GIMP, but it’s NO WAY a replacement for Photoshop in my mind!).

Then I turned to an IRC channel where I often stayed in the last few years from time to time. It’s the IRC channel of the BSDGroup.de. (The IRC channel is #bsdgroup.de on irc.freenode.org (6667) if you would like to join, but it’s mainly in German). I snapped some keywords on ZFS and RAIDz. Hm.. ZFS? Isn’t this the groundbreaking new filesystem from SUN Microsystems? Yes! And it was ported to FreeBSD. Wow! I like SUN, I like their hardware which is very structured build (ever installed hardware into a SUN? That’s very similar to the Mac Pro – just easy to do!) and of course I fight with Java on my Cisco’s notebook every day . No, I like SUN – they have humor and do cool commercials which you can check out on YouTube.

So, looks like RAIDz is what I’m looking for. A superstable filesystem that allows me to creat a kind of RAID5 on software base without having to buy one of those superexpensive cards. And it’s secure – more secure than any other filesystem. Woah! Sounds way too good, doesn’t it? Sounds perfect for my server project.

Here we go!

First, I grabbed the AMD64 ISO of FreeBSD 7.1 from their server, of course the “Boot Only” disc, which is similar to the “netinstall” of Debian. Just put it in your drive, boot and install. I did the setup several times in the past, so I remeber easily how to install it and what to take care about. The system harddisk is still on the old format, XFS, it’s on my Samsung 250GB SATA-II disk. Fine so far. After installing, I had to reboot the machine and login as root afterwards. Hey! No nano? What the hell…?

Ha! Getting software on your FreeBSD machine is something kinda easy. If you’ve a well powered CPU inside your machine, I would recommend you to go with the Ports collection which downloads the sources of the tool you would like to install and compile it right for your machine which gives every tool the perfect addiction to your machine. So for example installing nano looks like…

carter# cd /usr/ports/editors/nano
carter# make install clean
...
carter# rehash

Let me explain it. You join the directory by testing out where something is, or – more easy, by using whereis.

carter# whereis nano
nano: /usr/ports/editors/nano

Oh – by the way: Carter is the name of the server. My wife wants it

You can go on and install different things on your server the same way. I also installed bash, because the shell that came with FreeBSD is not so nice in my eyes, also it requieres this “rehash” command after every installation. Bash not

Ok – time to go and add some more diskspace to the computer:

In the picture above, you can see the server with final installation. From top to bottom I’ve added:

- Seagate 200GB SATA-II (Part 1/2 of RAIDz “backup”) ZFS
- Seagate 200GB SATA-II (Part 2/2 of RAIDz “backup”) ZFS
- WD “Green” 1000GB SATA-II (Part 1/3 of RAIDz1 “storage”) ZFS
- WD “Green” 1000GB SATA-II (Part 2/3 of RAIDz1 “storage”) ZFS
- WD “Green” 1000GB SATA-II (Part 3/3 of RAIDz1 “storage”) ZFS
- Samsung 250GB SATA-II (System) XFS

The server had 4Gigs of RAM, which is badly needed. Running RAIDz requieres at least 1Gig of RAM!

So, I started with creating the “zpool” (RAID storage) with the help of those two nice sites. I was really impressed that it told me after less than 10 seconds, that my RAIDz1 (storage) was created.

Let me show you how it looks after filling it with our content:

carter# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
storage 2.72T 456G 2.27T 16% ONLINE -

errors: No known data errors

That looks good, doesn’t it? Here’s the overview by running the “df -h” command to display the free/used capacities in your system:

carter# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ad10s1a 496M 245M 211M 54% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/ad10s1e 496M 12K 456M 0% /tmp
/dev/ad10s1f 216G 15G 184G 7% /usr
/dev/ad10s1d 4.7G 21M 4.3G 0% /var
storage 1.8T 304G 1.5T 17% /storage

Maybe you notice the difference between the sizes. I think it has something to do with the size units (1000 or 1024MByte = 1GB / 1000 or 1024GB = 1TB). I was shocked yesterday morning when I say TOO MUCH free space on this array But when I woke up (and this could take some time in the morning), I got it.

Next step was to create the other array. I used my 2x 200GB drives for a backup for the most important data. This is a kind of “RAID1″ array. Of course, I had to try to remove the data cable from one of those disks to see if it works. Perfect! I was able to copy all data with 1 missing drive on each array!

carter# zpool status
pool: backup
state: ONLINE
scrub: scrub completed with 0 errors on Wed Jan 28 17:45:12 2009
config:

errors: No known data errors

What you see here is the “scrub”. Scrub means to check the disk after a dropout of one disk. That’s why it’s recorded here.

carter# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
backup 372G 27.8G 344G 7% ONLINE -

carter# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ad10s1a 496M 245M 211M 54% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/ad10s1e 496M 12K 456M 0% /tmp
/dev/ad10s1f 216G 15G 184G 7% /usr
/dev/ad10s1d 4.7G 21M 4.3G 0% /var
backup 183G 14G 169G 8% /backup

I’ve removed the “other” entry everytime here (backup/storage) to make clear which is the one I’ve just spoken about.

So – cool, but how to get the data onto those RAIDz pools? Well, NOT the same way I used when I ran Windows or Linux servers. A painpoint regarding FreeBSD is the damn slow connection while using Samba3. So I decided to do a double-life. Using FTP as main option for transfering large files to the server, but using Samba site by site with FTP for small documents or sharing stuff to a network player for TV for example. Yes, it’s slow but funnily, in a test we did yesterday with a borrowd box, I was able to stream FullHD content to our plasma withour drops

So – that about the server. Other equipment has been purchased for the home office since I last mentioned it here. First of all, here are some nice shots from the office at the attic (we moved upstairs before christmas because of different reasons, one was, that I used to use 2 desks and not just one, just too much equipment to place ).

You can see the new case where the server is now built in on the right side of the desktop. Oh! By the way, I used Vista for a few days, but now I’m back on OSX here, and honestly – good to be back home . Vista is better than I thought, but – I played around too much with everything, so it became on overkill and a waste of time. I spend too much time on “trying” except get things done, for example blogging .

This is a recent shot from last Tuesday. I was watching John Chambers keynote at the Cisco Networkers 2009 over the VPN connection to the Cisco network. Very nice! I also loved the session of Brian Cox, a particle physican of the CERN. I’m very interested in astrophysics and this combination of Cisco and astrophysics – very win/win for me! Unfortunaly, I don’t see a recording of those session on the site . Need to ask around if they exist and if they are available for public viewing.

By the way equipment. I really hope noone of Cisco is now angry about me, but sorry, I couldn’t affored 400 Euros into an 8 Port SoHo switch from Cisco, so I bought this HP Procurve 1800-8G, as recommended by Stefan.

But, no worries, Cisco, I also bought a new Wireless router and this time, I bought a Linksys, which is part of Cisco

Stacked together and in operation mode, it looks like this (well, today all ports are used )

Before I forget, I twittered about thinking about installing FreeBSD on my X41, hey – got this supersweet 12″ ThinkPad X41 Tablet a few days ago and I’m really happy with it

That’s it for today, it’s 7 minutes before 9 and I need to urgently boot the Cisco notebook, have a lot things to get done today.

Have it good!

Ciao
Dennis

Tweet This Post Digg This Facebook Reddit