If you’re considering FreeNAS and look up forums.freenas.org, don’t be put off by the tone of the forum. I don’t know why the forum is so … well, snotty to people asking questions. There seems to me to be a “just do it my way and the Right Way, and pay for the high end hardware or don’t bother me” fog hanging over the whole thing. I have trouble biting my tongue for that.
It’s taken me some time and thought to reason this out.
1. ZFS is NOT a normal file system. It protects the entirety of your set of files very well indeed…
2. … but you pay for that. ZFS has some hidden pitfalls that can lose you all of your data at once, irretrievably. Step into one of the traps, and ALL of your data is gone, and probably not recoverable. Period.
3. The pitfalls are not the same pitfalls you’d find in normal file systems, so even experienced admins can hit them. And pitfalls by definition are not easy to see ahead of time.
It’s this sudden-loss-of-everything that seems to be driving the attitudes at forums.freenas.org, at least from what I can tell now.
There is good, solid technical advice there, but it’s often couched in a manner and style that is either unpleasant, patronizing, or downright offensive, to me at least.
I wish that the forum lent itself to some calmer “It’s better to do it this way than to [whatever is being advised against] because it exposes you to sudden and mysterious loss of ALL of your data in this way…”. It sometimes takes me some time to calm down and reason out why the advice came out the way it did.
But there is good technical content there, and good techie stuff is always worth working for, right?
I just finished the burn-in on the latest incarnation of the house data-furnace. I think I’m at the place where it would take a lot to move me off this one.
From the outside, it’s unimposing, and intended to be. It’s a 14″x14″x12″ cubish thing, black, and nearly featureless. It’s sitting beside it’s battery backup unit, and except for the power LEDs, there’s no good indication that it’s running other than the almost inaudible sound of air in the fans.
The case is a Fractal Node 804. I am thoroughly sold on this case. It’s small, it can hold up to eight drives easily and eleven if you’re determined, and has enough cooling with whispering fans to hold disk temps under 36C under full scrubbing activity for over an hour, and this with the stock fans. And it cost me $70 on sale. What’s not to love?
Well, maybe the fact that going over 11 drives is very, very hard and is best done with a different case. I figure that once I need more than the eight drives in RaidZ3 that’s possible in this case, I’ll have to get a lot more serious about data storage. At present, I could get eight 6TB disks in there theoretically, for 48TB of usable storage. That’s still a lot to me.
Inside are seven 3-TB drives in RaidZ3, giving me 12TB of usable storage. Using ZFS in RaidZ3 means that I can lose any THREE of those drives before I lose any data to disk failure. I got convinced to go with a server motherboard and processor in this version after using AMD and Asus boards in the previous versions. Server motherboards designed for using ECC and actually tested for it are much cheaper now, and so are server versions of intel processors. It has a Supermicro motherboard and a Xeon E3 and only uses 5W more than the AMDs and Asus at idle.
The screaming advantage here is that the Supermicro has an ethernet integrated power management interface (IPMI) that lets me turn it on and off as well as collecting thermal data and such over my house network, even with the machine in the attic, which is where it’s going eventually. That was, after all, what a lot of my previous wandering about with the network-button-pusher was about.
The latest release of FreeNAS that I’m using, 9.3, is a huge step forward, and works very well.
I mentioned giving up on OpenSolaris after Oracle’s hijinks.
FreeNAS, based on FreeBSD, works GREAT! I have a FreeNAS server up and running, and it fills all the needs I have.
I’ve also unwittingly had a few tests of its error tolerance. I had a new 1.5TB drive fail right in the middle of a several-Gb transfer. FreeNAS noted the problem, sent me an email to note that a disk drive was off-line, and DIDN’T EVEN STOP THE TRANSFER!
The transfer completed without a hitch. I know, I know, that’s how it’s supposed to work. But I’m always impressed when a reliability/error tolerance feature works as advertised in the real world.
Hats off to FreeNAS!
OK, I got a few requests for the parts list, so here it is:
Tr1 = wall-wart, 8Vac to 12Vac, or 10Vdc to 15Vdc output; this is very non-critical, as the other parts are designed to work with a wide range here.
DB1 = 1A/100V Diode Bridge Example: DF01S (Fairchild); this replaces the four diodes in the full wave bridge.
IC1 = 78L05 5V 100ma regulator
C1 =470uF/25V capacitor
C2 = 0.1uF 25V (or more) ceramic capacitor
D1 = 1A/100V (or more) diode
R1 = 1/4W 2.2K
IC2 = Microchips Technology TC54VN2702EMB713 (2.7V trip point, open drain)
R3 = 220 ohms 1/4W
ISO1 = transistor output optoisolator: example Fairchild FOD852
Notice that the specified TC54 part is an OPEN DRAIN output, so the schematic must be rearranged so that R3 attaches to the +5V supply to the TC54, and the cathode side of the LED in the optoisolator attaches to the output pin of the TC54. This is slightly different from the more generic schematic shown in the first posting.
The usual cautions apply here: you’re messing with AC power issues (although the wall wart keeps you out of the worst of it) and also messing with the power system on your computer. Don’t even try this unless you know and accept the risks of doing this, and have the skills to do this right. I have built one of these, and mine works. I may have made mistakes or omissions here; I don’t think so, but it’s always possible. You take responsibility for how your unit turns out if you build this.
I’ve given up on OpenSolaris. I flushed my OpenSolaris machine and am digging into FreeBSD and FreeNAS.
I get terribly frustrated at having given into what Oracle was trying to push me into doing all along. It feels to me as if Oracle is trying to make it difficult and unrewarding to use OpenSolaris so people will go away and quit bothering them if the people don’t want to pay Oracle’s price for Solaris use. I resent that bit of social-engineering/manipulation of me. I think it would have been more honest to just cut it off it.
That’s the way it seems to me, at least. Anyway, OpenSolaris is a non-starter to me at this point. I’ve moved the front end machine to FreeNAS, even with the remaining warts, and I will move the back end machine to FreeBSD when my testing of FreeNAS is complete. There are still some warts on FreeBSD/FreeNAS, but they’re probably not going to be open to manipulation of users the way I think that (Open)Solaris has/will be.
That’s my personal opinion. What you do is up to you.
I had one of those “Doh!!”, forehead slapping moments.
In the first Button-Pusher article I talked quite a bit about safety issues and the need to know what you’re doing with AC power line stuff – or wind up electrocuted. That was all straightforward and true.
But I think of these things like an engineer, not a hacker. The hacker side of me finally pulled through.
You don’t need to do all the AC safety mess if you have some widget that encapsulates all that for you. And we do, and it’s cheap and easy. It’s an AC to DC power adapter, widely know as a “wall wart”.
Just do the same circuitry after the rectifier diodes to do the internal stuff. But make the input to the computer side of this be a barrel style DC power connector. Plug an AC to DC wall wart in the controlled-AC power source, and plug the wall wart power connector into the barrel-plug socket on the computer. All done. No need to mess with AC power line. That’s all enclosed inside the wall wart. All that comes out is DC, and the dangerous bits have been pre-designed and pre-tested for safety and other things so that you never have to endanger yourself and others – at least any more than you do now. 😎
I find that there is a term for what I’ve been developing in the back of my head: Lights-Out Management. This is the idea that you can reset, power on/off, and in general tinker with and get back on line your servers. It’s a big deal at the commercial data centers, of course. I managed to back into the concept because I want a server in the attic and/or garage and am too lazy to climb/walk to it to reset/power/tinker with it.
I have a friend who is a professional data center administrator. He’s on call 24/7/365, and is always getting pages to fix a problem with the hundreds of servers in their center. To him, Lights-Out Management (or LOM as it’s called in the data biz) means the difference between fixing the problem from his home office and driving an hour each way to fix it.
Full LOM for a server includes power on/off and resetting the server as I’ve bumbled into, as well as remote access to low-level/BIOS setup, remote console support, and other things. This is so valuable that most real servers in the server world have some degree of it built in, perhaps for a fee.
But us home hobbyists can have some of that. I’ve outlined some of it in the Remote Button Pushers article. I’ve dredged up some additional info.
There is a card designed to add quite a lot of the LOM functions to an ordinary PC. It’s the PC Weasel Card. This thing gives you much of the LOM stuff, but to my way of thinking it misses a couple of them. HP/Compaq makes a similar card (google “RILOE”) that may work in other PCs.
First and foremost, it misses LAN connectivity, and it seems to not have a way to cut off/turn on the AC power. I think LAN manipulation is necessary (for me at least!) because I by definition already have LAN wires going to the server. I don’t necessarily also have RS232/phone wires going there. Sure, complete out-of-band control is nice, but it’s a home server and if I really had to, I could walk/climb to the server and mess with it even if the LAN is down.
In my mind, a good compromise for my home server would be to have a LAN connected card that could (a) “press” the reset button, (b) “press” the power button, (c) toggle AC power on/off if neither a nor b did what I needed. That’s how I stumbled into the IP Power thing – which by the way, works nicely.
There’s another option. I’ve recently found some ethernet-connected I/O servers. These are widgets that have an ethernet port at one end and some combination of digital I/O lines, RS232, analog I/O, USB, etc. on the other side. Most have some kind of embedded minimal web server so you can get to them by http and read/write the I/Os. The only problem seems to be that the producers of this ilk seem to want about $300-$1000 for them, a price comparable to a whole new server.
However, there are some that are under $100 if you’re willing to hack your own hardware. Here are two examples.
To make the PC Weasel card really useful, I could get to the RS232 port by using an ethernet-to-232 converter. Several of these exist. I think the makers of the Weasel are missing a bet here. And an ethernet to 232 that had digital I/O could then control AC power locally via an optoisolator controlling a relay, either magnetic or SSR.
I don’t really need the BIOS/frame buffer stuff that the Weasel has. I’m happy with remote power on/off and attachment to an SSH session. So what I need is a button pusher. I have one a suitable one on order.
A Korean company by the name of Wiznet makes an ethernet-to-232 card (WIZ107SR) and an ethernet-to-digital I/O card (WIZ220IO). What’s novel here is the low price: The WIZ220IO is $35 and the WIZ107SR is $22. I ordered the WIZ220IO and will be talking about how it works here. I don’t really need the serial interface, as I’d rather use the SSH session, but the serial interface would really neaten up the PC Weasel.
So, I’m hot on the trail of LOM for my home server. More later!
Well, mostly not. After some more time on google, I find that to make WOL work, you need
- A WOL-capable motherboard
- Setting the BIOS to enable WOL, or in newer boards “S5 on PME” wakeup events
- A WOL-capable LAN adapter; some add-in cards do this, some don’t; and some internal LAN adapters support it, some don’t
- A network device driver that supports WOL
It’s that last which Opensolaris fails on. The drivers may or may not have WOL inside them, depending on their provenance. And the WOL that may happen to be there may or may not work, or -worse!- may not work correctly, or reliably.
As I read when doing this research, the guys who form the Opensolaris cognoscenti do not think much of WOL. The general idea is that it’s at least useless, possibly worse. They vastly prefer wake from sleep/standby. Obviously this is a very broad generalization, and it’s what I inferred from a couple of hours of reading. Individuals may take issue with it.
I’m after remote wakeup and shutdown, as I mentioned in the post on remotely managed AC power. I had stopped work on the managed AC power button pushers while I investigated WOL. I’m now glad I did the AC button pusher work. It looks like I’ll need it.
I also found another suitable class of wakeup/shutdown devices. These go under the heading of embedded ethernet I/O devices. Some of them contain a minimal web server and can be run by a remote web browser. All of them have their own LAN port, and let you twiddle the I/Os remotely.
The application is obvious. These things can do the low-power button pushing and could conceivably have an AC power relay added. I’ll do more digging and see if this gets me closer to the poor-man’s “Lights-Out Adapter”, which is what the industry terms remote management setups.
Nuggets I found:
- The EON install CD will boot from an IDE interface CD drive with an IDE to SATA adapter plugged into it to attach to the SATA ports on a motherboard.
- But once EON has booted, it cannot mount that same IDE-to-SATA connected CD drive to use for completing the installation. The installation fails to complete, displaying the GRUB messages that it can’t find stage1 or stage2; that’s because they are on the CD during the installation process, and it can’t mount the CD drive with the odd plumbing I used for it.
- However, it works just fine if you put that same IDE CD drive into an external USB enclosure. EON can then both boot from and mount the USB connected CD drive.
- EON will happily install in a 2GB Compact Flash card which has been inserted into a Syba compact flash to IDE adapter. I stuck the SY-IDE2CF-DU adapter ($13, newegg.com) into the single IDE port on my motherboard. However, this eats up the only IDE port on my motherboard, like it would many others. That’s how I got into using an IDE CD drive with an IDE to SATA adapter for the install CD.
- EON saves all the configuration that you set up back on the boot disk you installed to. Or at least it does if you update the boot image with the “updimg.sh” after you do the config changes.
- … and you have to type in “/usr/bin/updimg.sh /mnt/eon0/boot/x86.eon” and not just “updimg.sh”. Doh! I was lucky that the command told me what to do as “correct usage”.
- I still don’ t have the DHCP stuff running, so I just set the IP address to a fixed value inside my home net. That got me running on ssh console logins.
- … but then I got strange errors and Apache going into maintenance mode. I fixed that by editing the hosts file to include the server name and the IP address I’d chosen. Now I can stick the server address into a browser in my home net and it replies with the thoughtfully provided message “It works!”
The part that’s worked solidly since the first try was zfs setup and operation and cifs shares. Worked first time. I hope this saves you some time.
[Note: as of 5-23-2011 I have decided to ditch OpenSolaris entirely based on Oracle’s new directions for Solaris and OpenSolaris. My data is too important to depend on what I perceive from Oracle.]
You can spin up your disks one at a time if your hardware allows it. Making your hardware allow it can be tricky, but worthwhile. You can use a much smaller power supply.
Low power is a virtue in a home server, just like it is for a commercial site server. When a disk spins up, it temporarily uses as much as five to ten times as much power as it does when it’s operating normally.
That presents a problem when you turn on the power supply. I have eight disks in my server at the moment, six 750GB 3.5″ devices and two 40GB 2.5″ devices. The main array is the six 3.5″ devices, which get most of their power from +12V. At start up they can pull as much as 3A (36W) of +12V power each. That’s 18A of +12V – 216W. After spinning up, the disks settle to 6W each, or three amps of +12V for the whole array.
The rest of the system uses +12V to power the processor, memory, and other things, so booting the entire system takes even more. When the system gets past this power-on-peak, it settles to about 100W total. There is a big pulse of power to get the mess started.
No problem, just buy a big power supply right? Actually, that is a problem. Power supplies waste power making the power you use. The power supply efficiency is the ratio of the power you get out of it to the power you put into it from the AC power line. It’s only recently that you could get power supplies which stated, and lately, certify their efficiency. The best are rated for “80-plus” which stands for “better than 80 percent”. These especially-good power supplies eat 100W of AC power line power for every 80W of power they deliver to your computer.
What that doesn’t tell you is that the losses are usually fixed, or at least do not decrease much with lower power out. A 100W-rated 80+ power supply “eats” 125W from the AC power line to make 100W of output power, the 25W being the efficiency penalty. A 500W rated 80+ power supply wastes five times as much, 125W, to make 500W, and it probably wastes 100W all the time, even if the load is only getting 100W. And the 80-plus units are the good guys. Lower power supply efficiencies only make the wasted power worse.
There is therefore a big plus in terms of less wasted electricity and heat generation if you can buy a smaller power supply for your server. But we already figured out that we have to buy a power supply to withstand the power-on peak in power needs. The power supply probably loafs after that.
Staggered spin-up is one trick to getting the peak power down, letting you buy a smaller, less wasteful power supply. If you could power up only one disk at a time, you could start your system with a much smaller power supply. In my system, the measured idle power from the AC line is about 105W. Each disk pulls 6W at idle, but 36W at spinup.
Theoretically, I could get the system to start up successfully with a power supply rated at 105W plus 30W for the peak for one disk drive to spin up if i could ensure that only one disk would start at a time. A 150W power supply would be fine for that! Not only cheaper, more economical to run and “greener”.
To get to staggered spin up, you have to enable both hardware and software to make it happen. Opensolaris is great for that, because you can enable staggered spinup. But how about the hardware?
It turns out that some SATA drives (notably from WD) have a jumper on the drive for enabling and disabling staggered spin up. Others (notably Seagate) use pin 11 of the SATA power connector for enabling staggered spinup. Disabling staggered spin up happens when pin 11 is grounded, enabling happens when it’s open. It took me a few days to find that out.
When I could not make staggered spin up work, I dug in and found out that my power supply cable connectors have pin 11 internally connected to pins 10 and 12 – they are hard wired to make Seagate drives not do staggered spin up! The only way to make Seagates do staggered spin up was to commit surgery on either the disks or the power connectors to isolate pin 11. Then they spun up only when the software told them to.
As usual, some of the more useful bits are buried quite deeply. But if you look, you can sift out some power and money savings in your server.