R.G. Keen's Blog

I’ve given up on OpenSolaris

Posted in Opensolaris Server by rgkeen on May 23, 2011

I’ve given up on OpenSolaris. I flushed my OpenSolaris machine and am digging into FreeBSD and FreeNAS.

I get terribly frustrated at having given into what Oracle was trying to push me into doing all along. It feels to me as if Oracle is trying to make it difficult and unrewarding to use OpenSolaris so people will go away and quit bothering them if the people don’t want to pay Oracle’s price for Solaris use. I resent that bit of social-engineering/manipulation of me. I think it would have been more honest to just cut it off it.

That’s the way it seems to me, at least. Anyway, OpenSolaris is a non-starter to me at this point. I’ve moved the front end machine to FreeNAS, even with the remaining warts, and I will move the back end machine to FreeBSD when my testing of FreeNAS is complete. There are still some warts on FreeBSD/FreeNAS, but they’re probably not going to be open to manipulation of users the way I think that (Open)Solaris has/will be.

That’s my personal opinion. What you do is up to you.


Remote Power Management, Part Four: A Safer Button Pusher

Posted in Opensolaris Server, Power Considerations, Remote Management by rgkeen on May 16, 2010

I had one of those “Doh!!”, forehead slapping moments.

In the first Button-Pusher article I talked quite a bit about safety issues and the need to know what you’re doing with AC power line stuff – or wind up electrocuted. That was all straightforward and true.
But I think of these things like an engineer, not a hacker. The hacker side of me finally pulled through.
You don’t need to do all the AC safety mess if you have some widget that encapsulates all that for you. And we do, and it’s cheap and easy. It’s an AC to DC power adapter, widely know as a “wall wart”.

Just do the same circuitry after the rectifier diodes to do the internal stuff. But make the input to the computer side of this be a barrel style DC power connector. Plug an AC to DC wall wart in the controlled-AC power source, and plug the wall wart power connector into the barrel-plug socket on the computer. All done. No need to mess with AC power line. That’s all enclosed inside the wall wart. All that comes out is DC, and the dangerous bits have been pre-designed and pre-tested for safety and other things so that you never have to endanger yourself and others – at least any more than you do now. 😎

Lights-Out Management For Home Servers

Posted in Opensolaris Server, Remote Management by rgkeen on May 15, 2010

I find that there is a term for what I’ve been developing in the back of my head: Lights-Out Management. This is the idea that you can reset, power on/off, and in general tinker with and get back on line your servers. It’s a big deal at the commercial data centers, of course.  I managed to back into the concept because I want a server in the attic and/or garage and am too lazy to climb/walk to it to reset/power/tinker with it.

I have a friend who is a professional data center administrator. He’s on call 24/7/365, and is always getting pages to fix a problem with the hundreds of servers in their center. To him, Lights-Out Management (or LOM  as it’s called in the data biz) means the difference between fixing the problem from his home office and driving an hour each way to fix it.

Full LOM for a server includes power on/off and resetting the server as I’ve bumbled into, as well as remote access to low-level/BIOS setup, remote console support, and other things. This is so valuable that most real servers in the server world have some degree of it built in, perhaps for a fee.

But us home hobbyists can have some of that. I’ve outlined some of it in the Remote Button Pushers article. I’ve dredged up some additional info.

There is a card designed to add quite a lot of the LOM functions to an ordinary PC. It’s the PC Weasel Card. This thing gives you much of the LOM stuff, but to my way of thinking it misses a couple of them. HP/Compaq makes a similar card (google “RILOE”) that may work in other PCs.

First and foremost, it misses LAN connectivity, and it seems to not have a way to cut off/turn on the AC power.  I think LAN manipulation is necessary (for me at least!) because I by definition already have LAN wires going to the server. I don’t necessarily also have RS232/phone wires going there. Sure, complete out-of-band control is nice, but it’s a home server and if I really had to, I could walk/climb to the server and mess with it even if the LAN is down.

In my mind, a good compromise for my home server would be to have a LAN connected card that could (a) “press” the reset button, (b) “press” the power button, (c) toggle AC power on/off if neither a nor b did what I needed. That’s how I stumbled into the IP Power thing – which by the way, works nicely.

There’s another option. I’ve recently found some ethernet-connected I/O servers.  These are widgets that have an ethernet port at one end and some combination of digital I/O lines, RS232, analog I/O, USB, etc. on the other side. Most have some kind of embedded minimal web server so you can get to them by http and read/write the I/Os. The only problem seems to be that the producers of this ilk seem to want about $300-$1000 for them, a price comparable to a whole new server.

However, there are some that are under $100 if you’re willing to hack your own hardware. Here are two examples.

To make the PC Weasel card really useful, I could get to the RS232 port by using an ethernet-to-232 converter. Several of these exist. I think the makers of the Weasel are missing a bet here. And an ethernet to 232 that had digital I/O could then control AC power locally via an optoisolator  controlling a relay, either magnetic or SSR.

I don’t really need the BIOS/frame buffer stuff that the Weasel has. I’m happy with remote power on/off and attachment to an SSH session. So what I need is a button pusher. I have one a suitable one on order.

A Korean company by the name of Wiznet makes an ethernet-to-232 card (WIZ107SR) and an ethernet-to-digital I/O card (WIZ220IO). What’s novel here is the low price: The WIZ220IO is $35 and the WIZ107SR is $22. I ordered the WIZ220IO and will be talking about how it works here. I don’t really need the serial interface, as I’d rather use the SSH session, but the serial interface would really neaten up the PC Weasel.

So, I’m hot on the trail of LOM for my home server. More later!

Curses – Opensolaris doesn’t support Wake On Lan!

Posted in Opensolaris Server, Power Considerations, Remote Management by rgkeen on May 12, 2010

Well, mostly not. After some more time on google, I find that to make WOL work, you need

  • A WOL-capable motherboard
  • Setting the BIOS to enable WOL, or in newer boards “S5 on PME” wakeup events
  • A WOL-capable LAN adapter; some add-in cards do this, some don’t; and some internal LAN adapters support it, some don’t
  • A network device driver that supports WOL

It’s that last which Opensolaris fails on. The drivers may or may not have WOL inside them, depending on their provenance. And the WOL that may happen to be there may or may not work, or -worse!- may not work correctly, or reliably.

As I read when doing this research, the guys who form the Opensolaris cognoscenti do not think much of WOL. The general idea is that it’s at least useless, possibly worse. They vastly prefer wake from sleep/standby. Obviously this is a very broad generalization, and it’s what I inferred from a couple of hours of reading. Individuals may take issue with it.

I’m after remote wakeup and shutdown, as I mentioned in the post on remotely managed AC power. I had stopped work on the managed AC power button pushers while I investigated WOL. I’m now glad I did the AC button pusher work. It looks like I’ll need it.

I also found another suitable class of wakeup/shutdown devices. These go under the heading of embedded ethernet I/O devices. Some of them contain a minimal web server and can be run by a remote web browser. All of them have their own LAN port, and let you twiddle the I/Os remotely.

The application is obvious. These things can do the low-power button pushing and could conceivably have an AC power relay added. I’ll do more digging and see if this gets me closer to the poor-man’s “Lights-Out Adapter”, which is what the industry terms remote management setups.

Lessons from Installing EON ZFS Server

Posted in Opensolaris Server by rgkeen on May 8, 2010

Nuggets I found:

  • The EON install CD will boot from an IDE interface CD drive with an IDE to SATA adapter plugged into it to attach to the SATA ports on a motherboard.
  • But once EON has booted, it cannot mount that same IDE-to-SATA connected CD drive to use for completing the installation.  The installation fails to complete, displaying the GRUB messages that it can’t find stage1 or stage2; that’s because they are on the CD during the installation process, and it can’t mount the CD drive with the odd plumbing I used for it.
  • However, it works just fine if you put that same IDE CD drive into an external USB enclosure. EON can then both boot from and mount the USB connected CD drive.
  • EON will happily install in a 2GB Compact Flash card which has been inserted into a Syba compact flash to IDE adapter. I stuck the SY-IDE2CF-DU adapter ($13, newegg.com) into the single IDE port on my motherboard. However, this eats up the only IDE port on my motherboard, like it would many others. That’s how I got into using an IDE CD drive with an IDE to SATA adapter for the install CD.
  • EON saves all the configuration that you set up back on the boot disk you installed to. Or at least it does if you update the boot image with the “updimg.sh” after you do the config changes.
  • … and you have to type in “/usr/bin/updimg.sh /mnt/eon0/boot/x86.eon” and not just “updimg.sh”.  Doh! I was lucky that the command told me what to do as “correct usage”.
  • I still don’ t have the DHCP stuff running, so I just set the IP address to a fixed value inside my home net. That got me running on ssh console logins.
  • … but then I got strange errors and Apache going into maintenance mode. I fixed that by editing the hosts file to include the server name and the IP address I’d chosen. Now I can stick the server address into a browser in my home net and it replies with the thoughtfully provided message “It works!”

The part that’s worked solidly since the first try was zfs setup and operation and cifs shares. Worked first time.  I hope this saves you some time.

[Note: as of 5-23-2011 I have decided to ditch OpenSolaris entirely based on Oracle’s new directions for Solaris and OpenSolaris. My data is too important to depend on what I perceive from Oracle.]

Staggered Disk Spin-up for Home Servers

Posted in Opensolaris Server, Power Considerations by rgkeen on May 2, 2010

You can spin up your disks one at a time if your hardware allows it. Making your hardware allow it can be tricky, but worthwhile. You can use a much smaller power supply.

Low power is a virtue in a home server, just like it is for a commercial site server. When a disk spins up, it temporarily uses as much as five to ten times as much power as it does when it’s operating normally.

That presents a problem when you turn on the power supply. I have eight disks in my server at the moment, six 750GB 3.5″ devices and two 40GB 2.5″ devices.  The main array is the six 3.5″ devices, which get most of their power from +12V. At start up they can pull as much as 3A (36W) of +12V power each. That’s 18A of +12V – 216W.  After spinning up, the disks settle to 6W each, or three amps of +12V for the whole array.

The rest of the system uses +12V to power the processor, memory, and other things, so booting the entire system takes even more. When the system gets past this power-on-peak, it settles to about 100W total. There is a big pulse of power to get the mess started.

No problem, just buy a big power supply right? Actually, that is a problem. Power supplies waste power making the power you use. The power supply efficiency is the ratio of the power you get out of it to the power you put into it from the AC power line. It’s only recently that you could get power supplies which stated, and lately, certify their efficiency. The best are rated for “80-plus” which stands for “better than 80 percent”. These especially-good power supplies eat 100W of AC power line power for every 80W of power they deliver to your computer.

What that doesn’t tell you is that the losses are usually fixed, or at least do not decrease much with lower power out.  A 100W-rated 80+ power supply “eats” 125W from the AC power line to make 100W of  output power, the 25W being the efficiency penalty. A 500W rated 80+ power supply wastes five times as much, 125W, to make 500W, and it probably wastes 100W all the time, even if the load is only getting 100W. And the 80-plus units are the good guys. Lower power supply efficiencies only make the wasted power worse.

There is therefore a big plus in terms of less wasted electricity and heat generation if you can buy a smaller power supply for your server. But we already figured out that we have to buy a power supply to withstand the power-on peak in power needs. The power supply probably loafs after that.

Staggered spin-up is one trick to getting the peak power down, letting you buy a smaller, less wasteful power supply. If you could power up only one disk at a time, you could start your system with a much smaller power supply. In my system, the measured idle power from the AC line is about 105W. Each disk pulls 6W at idle, but 36W at spinup.

Theoretically, I could get the system to start up successfully with a power supply rated at 105W plus 30W for the peak for one disk drive to spin up if i could ensure that only one disk would start at a time.  A 150W power supply would be fine for that! Not only cheaper, more economical to run and “greener”.

To get to staggered spin up, you have to enable both hardware and software to make it happen. Opensolaris is great for that, because you can enable staggered spinup. But how about the hardware?

It turns out that some SATA drives (notably from WD) have a jumper on the drive for enabling and disabling staggered spin up. Others (notably Seagate) use pin 11 of the SATA power connector for enabling staggered spinup. Disabling staggered spin up happens when pin 11 is grounded, enabling happens when it’s open. It took me a few days to find that out.

When I could not make staggered spin up work, I dug in and found out that my power supply cable connectors have pin 11 internally connected to pins 10 and 12 – they are hard wired to make Seagate drives not do staggered spin up! The only way to make Seagates do staggered spin up was to commit surgery on either the disks or the power connectors to isolate pin 11. Then they spun up only when the software told them to.

As usual, some of the more useful bits are buried quite deeply. But if you look, you can sift out some power and money savings in your server.

Remote Power Management, Part Three: Button Pusher Circuits

Posted in Opensolaris Server, Power Considerations, Remote Management by rgkeen on April 28, 2010

I’ve designed a couple of circuits which implement the button pusher circuits for remotely pressing the power-on and/or reset buttons on an ATX style machine from a LAN controlled AC power outlet.

The circuits are trivially easy to make if you’re an electronics hardware type. I’m doing PCB layouts of both circuits for myself and can make those available for you to use if there’s any interest.

Let’s do the warnings up front. These circuits necessarily involve connections to the AC power line because that’s how they work – they convert the AC power line coming on to an isolated contact inside a controlled machine. There’s no way around that. The circuits are easy but any time you even get physical access to AC power line wiring, there is danger of electrocution, electrical damage to property and starting fires. Living in an electrically-powered world like we do the rule for everyone has to be  that if you don’t know how to do it safely, don’t start. So read that safety warning in the center of the picture and follow it. It’s YOUR responsibility.

I apologize for the legalism, but in a lawsuit-mad society, I cannot be personally responsible for your actions. You have been warned. Act accordingly.

With that bit of nastiness out of the way, let’s get on with doing what we started out to do.

Circuit 1 is the no-transformer version. The amount of electrical power needed in faking a button activation is tiny. So the first version uses a capacitor (C1) to limit the amount of current the AC power line provides to the rest of the circuit, and it does it in a way that is very power-efficient. The rest of the circuit takes this little trickle of current, makes just enough DC voltage out of it to run the integrated circuit. The integrated circuit is a three pin device that does only one thing – it senses whether its power supply is big enough, and when it is, it pulls its output pin high.  High in this case is about 5Vdc, and that’s enough for the resistor R3 to allow enough current through the LED of the optoisolator ISO1.

This causes the isolated phototransistor on the isolator’s output to be fully saturated and look like closed contacts to the ATX power supply. If you connect the isolators output across the “power switch” contacts in the contacts of the motherboard you’re using, then it will look just like you’ve pressed the front panel switch to the motherboard, and this will activate the power supply on/off just like you pressing the button.

Two remote button pusher circuits

Button Pusher #2 does exactly the same thing on the computer side. What’s different is the way that the power supply for the output is generated. Instead of using resistors and capacitors to make the voltage needed by the reset IC, it uses a transformer to both isolate the AC power line and to change the power line voltage down to the few volts needed for the reset IC. This voltage is rectified, filtered, and then regulated by a tiny three terminal regulator. This circuit is bigger and heavier than Button Pusher #1, but not much. I was surprised to find that there are now tiny transformers which will do this job, and that they are not expensive.

I’m not going to clutter this posting up with parts lists and so on, because I suspect that very few people will ever actually build one of these. If you want to really build one of these, instead of just reading about it, leave a comment so we can get in touch one on one.

Remote Power Management for Home Servers

Posted in Opensolaris Server, Power Considerations, Remote Management by rgkeen on April 26, 2010

Shortly after I got my first Opensolaris server up and running, I discovered that I never wanted to see it again – servers should be usable but invisible as well as silent!

Back to google. A long series of searches led me to some conclusions and solutions.

  1. LAN managed power cycling is a Very Good Thing if you are not able to walk over to your server to restart/reboot it. It will be needed. My neighbor makes his living doing system admin for a commercial data center with hundreds of systems in racks. He has to drive in to work in the middle of the night often enough even with industrial-strength remote management setups as it is.
  2. Hard power-off is potentially life-threatening to your data. Soft power off is much kinder. Hard power off should be reserved for situations where the system is no longer listening to your instructions and there is no option. Again, my neighbor who lives and breathes remote support confirms that he’ll do pretty much anything to avoid hard power-off on a critical server. His opinion is that hard power-off forces you to consider server CPR as a real possibility.
  3. LAN managed power cycling is moderately expensive. And I’m – well, I prefer the term “economically prudent” to cheap.

I can hear you wondering why I didn’t just set up Wake On LAN (WOL). That would actually be a great thing to do except for two issues.  First, not all motherboards support WOL. My first motherboard, the M3A78-CM which is otherwise great, does not. Worse yet, it does not clearly tell you that in the manual. I had to send a tech support request to the maker to find that out unambiguously.

Then again, WOL is for when things are all working OK. I can always tell my machine to shut down if it’s listening to me. But if it’s too busy contemplating its own navel and re-counting its toes to be bothered to listen, there ought to be some way to make it listen. I think I found it.

A short series of conclusions came up in the searches.

There is a sweet spot for LAN managed AC power. It’s the IP Power 9258 series. This is a small box with an ethernet port, four managed AC outlets, and an internal web server to flip the managed outlets’ states. Here’s a copy of the manual.

The current manufacturing state of the unit seems to be in question. A google search turns up many places which say out of stock and no longer available. However, there are also many places which have them, including a seller on ebay for $80. It’s in the mail to me as I write this, so we’ll know soon.

I found another technique called “Whack-on-LAN”, which uses a modified ethernet NIC which has a WOL connector to do what amounts to  pressing the reset button over the LAN. The article describing this is here. In essence, you convert a cheap LAN card to a remote-controlled button pusher and hit the reset button. The only issues with this trick is that the LAN card lives inside the server and is dependent on the server power supply and BIOS/operating system making it happy so it can do the LAN remote control trick.

So there’s some interesting stuff. In my mind, it plays out like this:

  • I want to be able to tell a non-WOL system to fire up. A LAN-controlled button pusher would be great for that.
  • Any system that’s working properly will shut down on its own.
  • Any system that’s not working properly would have to be hard-reset.
  • If the hard-reset doesn’t work, it needs hard power-off cycled as a last resort.
  • There are advantages to an integrated power management system even for a board that supports WOL.

Here’s the solution I came up with. Use the 9258 to control AC power. I really only need one of the controlled AC outlets to run the system. That leaves me three other outlets which can be viewed as a form of LAN controlled logic outputs – that is, AC power = logical true, no AC power = logical false. What is needed to get all of what I need for deluxe remote power management for my server is to make those AC power logic levels into isolated contact closures, which then can interface directly into the normal circuitry of the industry-standard ATX form factor personal computer stackup.

Did I mention that I spent many years doing analog and digital circuit design? The two obvious answers are either to use an AC-activated relay with a power-line rated primary run from a controlled outlet, or an electronic circuit which senses AC power and does an isolated “contact closure” similar to the transistor switch use in “Whack-on-LAN”. There are many optoisolators available with 4kV isolation from the LED driver on the primary side to a transistor output on the secondary side which will safely do this job.

I’ve done a quick design of both of these approaches, and I’ll document those here in a future post.

ECC for a Home Server?

Posted in Data Integrity, Opensolaris Server by rgkeen on April 25, 2010

One major decision in building a home server is whether you will use ECC (Error Checking and Correcting) RAM or not.

To save myself some typing I suggest you look at Wikipedia’s entry on ECC memory for some background.

Your main memory is dynamic RAM chips, and they do get soft errors. The error rate may vary between tragic (one error per GB per month) to trivial (one error per GB per century). If you’re lucky and get one of the 1/century setups, great. With my luck, I normally get stuff from the other end of the distribution. So I demand both that my server have ECC and actually use it productively, which are two different things. I suggest a look at Robin Harris’ blog on memory errors for some more background. Errors do happen. They are sometimes sly and unnoticeable until they have killed something.

I split servers into three classes in my mind: trivial servers, moderate servers, and major servers.  Trivial servers are a handy bit bucket, a place to store some stuff. No significant attention is paid to data integrity or future expansion. Moderate servers differ from major servers in that they have features for data integrity and future expansion, but are very parsimonious about how much money is spent on the whole setup to get these advantages. Major servers are a data-center creature, with planning, support staff, and so on. Personal home servers are usually moderate servers.

In my mind, I think that trivial servers do not have ECC, moderate servers do, and major servers must.

This is a lurking Big Deal in server decisions because ECC is not that available in over-the-counter motherboards. The X86 motherboard market has split into two major lines: desktops and servers. Desktops are what we normally think of as a motherboard when we plan a new build: pick a processor, pick a CPU, grab some memory, and bango, new system. Not all that long ago, the normal run of the mill motherboard chipset could and might support ECC. The difference is, unsurprisingly, money. Desktop motherboards are a very, very cost sensitive market, and a penny saved is a sale made. Server motherboard buyers are both informed enough and motivated enough to spend a bit more for data reliability.

Intel-provided chipsets have done the same split as the motherboard market. Their latest family of desktop motherboard chipsets for the i5 and i7 families definitely do not support ECC. If you want ECC and demand Intel CPUs and chipsets, you’re going to have to click that “server motherboards” link and pay more for them. According to my frantic googling, the premium runs from about US$200 if you are very, very thorough to as much as $2K.

AMD’s latest desktop processors and chipsets do support ECC, and this leads me to one of my choices – use AMD CPUs and find a motherboard that supports ECC. The added complication is that even here, penny-pinching has struck, and many motherboards which have AMD CPUs and chipsets which nominally support ECC have removed either the BIOS support for ECC or the actual PCB traces that let it work, or both.

This lets them tell you that the motherboard is compatible with ECC memory without actually giving you ECC. I refer to this condition as ECC-memory tolerant, not enabled.

The key seems to be looking in the motherboard manual for a way to enable/disable/set up ECC in the BIOS memory setup section.  Note that I consider that if the motherboard vendors will not let you download the user manual before you buy it, you should not buy that motherboard. Full stop.

So you need to decide: are you building a trivial server or a moderate server?

A trivial server is a perfectly valid thing to build, as long as you have looked at its limitations and are OK with them. It is perfectly reasonable to want, say, a very-low-power server for minimal cost with limited data storage as long as you know and accept that you will be getting soft memory errors and soft disk errors over the long haul.

I want a home server that gives me more data integrity. I am willing to spend some more money to get that, but would like to keep the premium down to as low as possible. In my personal evaluation system, that means I need ECC memory. That drives the decisions: AMD low-end CPUs and chipsets support ECC more inexpensively than the semi-equivalent Intel ones. And within the AMD motherboard space, some motherboards support explicit BIOS options for ECC enablement.

As a simplifying matter, all ASUS AM2, AM2+, and AM3 motherboards seem to have ECC support in BIOS, as well as statements about support for ECC and non-ECC DRAM. Only some Gigabyte motherboards with the same chip set may support ECC; they make statements like “ECC is supported only for CPUs which support ECC”, which kind of implies but does not make clear about their support. There are a few other boards, notably a Biostar one or two with AMD chipsets that do, I think. Out of this space, if you’re going with Opensolaris, you need to check the Opensolaris forums and Hardware Compatibility List (HCL) and pick one which clearly has support.

I picked an ASUS M3A78-CM (appx US$80 in Jan 2010) on this basis. My server motherboard came up and installed first time, no errors, and ran Opensolaris. This motherboard has been discontinued – probably why I got a great deal on it! – but the newer M4A785 boards are a very reasonable alternative.

I think ECC is mandatory for one of my primary objectives – data integrity.
[Note: as of 5-23-2011 I have decided to ditch OpenSolaris entirely based on Oracle’s new directions for Solaris and OpenSolaris. My data is too important to depend on what I perceive from Oracle. I feel like I personally would be better off trusting my data to a system less susceptible to manipulation.]

Storage Needs in a Home Server

Posted in Opensolaris Server by rgkeen on April 25, 2010

You need to decide ahead of time how much storage you will reasonably need. In a non-trivial server, the amount of  storage to be included is a major consideration, as well as the major up-front expense. In a trivial server, the storage is secondary to the rest of the system.

You will need to include all the storage space you reasonably think you’ll need, plus some expansion room, plus additional storage for the redundancy that will make your storage reliable.

Redundant storage is a price of data reliability. Redundant storage in the form of mirrored disks, checksum-containing disks, or spare disks, or all of these, is what gives your server the ability to recover from failures.

Unless you have a very small server objective, it is well to consider the amount your server storage can be expanded without a major and possibly expensive rebuild.

I realize in rereading this that it sounds too simplistic for many people to take seriously. However, remember that people are data packrats. The largest amount of storage that you can imagine using will be overwhelmed in a few years.

I recall an illustration from the pre-personal-computer days when centrally-managed data centers were all the computing that was available. A major university was dealing with storage issues on its central computers, and this was in a time when one megabyte of main memory cost US$1M and the US dollar was worth about five times what it is today (2010). The system admins installed an access counter that incremented a counter for each file in the system every time it was accessed. The average number of accesses per file was about 1.1, and that included the access which created the file.  And that included the system files, compilers, utilities, that every user accessed all the time.  The user body was, in effect, using the storage pretty much as write-only-memory, storing stuff they never went back to.

Trust me – you need to think about expansion of your storage system.