This post is going to be somewhat of a “link dump” for me of some pages that I’ve been perusing lately. After playing with RT (request tracker) – I added a few ticket items for the home network. Now, if you’ve been a longtime reader and sorted through ALL of these posts here you’ll know that I’ve made use of software raid on the home systems. Why? Linux software raid seems fairly reliable (so far – 2 years +). It doesn’t depend on a specific piece of hardware. In short IF the worst happens and the array fails I should be able to retrieve data from an individual drive more easily than if it were hardware raid. I’m using Raid Level 1 (cloning/mirroring) and ext3 is the filesystem on top. I’ve had some slight problems with one drive in both the Desktop and server arrays and both arrays had been degraded for some time. My goal was ultimately redundancy and to eliminate the disruption that hard drive failures have given over the last few years.

When push comes to shove I’m cheap and hadn’t looked at replacing the drives yet. (Potentially a problem especially since I was down to 1 drive for 2 out of 3 arrays.) So, I was running a risk for several months. The main reasons I’ve procrastinated is 1) I’m cheap and 2) I’m busy. The other reason though is the two arrays were made of 400gb sata drives which are now hard to find. 500 gb is the new standard size. Now, I don’t want to waste space so… it’s been a bit of a mental puzzle thinking about replacing one, growing the partition size and then replacing the other to pass along to the second system.

So….. long story short. I was able to re-add the drives to their respective arrays after researching the best way to grow a mirrored array onto larger devices. Why? When the arrays degraded they did so for pending sector reads. I’ve tested the drives via smart and they seem okay. So, I tried re-adding and so far all seems good.

Anyway, here’s the link dump:

Good reminder – http://www.freddenny.com/UNIX/linuxRAID.html – making raid volumes bootable when moved.

Resizing a raid 1 system partition – http://mkfblog.blogspot.com/2007/11/resizing-raid1-system-partition.html

Looks like gparted supports software raid now – http://gparted.sourceforge.net/news.php

Just some notes for a reminder.

RAID is not a substitute for backups. If you accidentally delete a file, it’s still GONE. If you have corruption on one drive (or via the bus) your data is still GONE. Schedule backups of important data.

In sum – replacing drives in a software raid array is quite simple – especially if they’re the same size. Moving to larger drives and not wasting space is a bit more complicated and is a bridge I’ll have to cross at some point.

As a side note, I discovered that I had two more drives of equal size (2 160GB drives) in one system that I had originally intended to mirror those as well, but they needed data cleanup. I spent some time cleaning up the data on one then deleted it’s partitions – created a new partition of the linux auto raid type and create a new raid device with that drive and a missing drive. After that, I created the ext3 filesystem and moved the files over from the other drive. Then I blew away the partition on the 2nd drive and re-partitioned that to be a linux auto raid partition. Then I added it to the array and all synced well.

So, now – all of my 24/7 uptime drives are mirrored and I’ve got some research notes for how to move things around to larger drives when the time comes. I still should add a to-do item to check and see how many more sata drives I could squeeze into my desktop or server – it might be nice to have a hot spare.

Once upon a time I wrote of manually extracting a file from a mondorescue backup. Sometimes it’s just easier to do that, than have mondorescue go through 30 some cds or dvds just to find one file. (Not to mention the fact that the iso’s aren’t burned to disc but are just stored on a usb HD.) So, I’ve just had the opportunity to try to restore one of the “biggiefiles”. Mondoarchive attempts to split huge files up into smaller slices for archiving. The size is something that I think is configurable, but I haven’t made any changes from the default.

The first trick is figuring out which set of slice-0000105-0000.dat files to hunt for. Find the list of biggiefiles (usually in the archives folder called biggiefiles.txt) grep the file looking for your filename, but we will need the line number of the file. In my case I did `grep -n myfilename biggiefiles.txt` and found that the file I was looking for was at line 106.

So, I started looking and found that the slice-0000106-0000.dat file reported a different filename when I used head to read the beginning of it. So, I looked at slice-0000105-0000.dat…. aha! Our line 1 in biggiefile.txt get’s to be slice-0000000-0000.dat So we’ll always be one number off from the line number.

Okay – so in my case there were about 17 slices… ranging from slice-0000106-0000.dat to slice-0000106-0017.dat I should mention that many of these slices are bz2 compressed – so the first step is to uncompress them all bunzip2 *bz2 in the folder I was using as a temp folder for this worked.

Next, I just tried cat’ting them all together `cat slice*.dat >all_files.dat` But the resulting file was read as corrupt (I was restoring an access database. So, after a bit of looking at the mondo-archive code…. I tried something a bit different. I renamed the first slice (ending in 0000.dat) to something outside of our slice- count sequence -so I just called it outside.dat and then cat’ted the rest of them together using the same command as above. Why ? The 0000.dat file apparently is written as a header with just the filename and perhaps other information that mondoarchive uses (size?) I really don’t know what else, but I had noticed that the header on the 0001.dat was similar to the header on my real .mdb files that were floating around.

Anyway – I’m making a note of it here in case I need it again. Probably the best thing I can suggest to start with is to create a folder to work on these files outside of your backup structure. And by all means COPY instead of move the files over into it.

Recently I had the opportunity to work on an Acer Aspire One netbook. It’s rare that I get the chance to work on a linux desktop system for a client. Most of my desktop users that I support run Windows (although I primarily run linux on my machines.) I get to do a fair amount of server level work with linux variants, but this one, being a desktop install, intrigued me.

The issue was that the buttons on the trackpad didn’t work. She also indicated that she had tried a usb mouse and those buttons didn’t work either. As I looked at it, I had already spent some time browsing through bug reports getting ideas on what might be the issue. It behaved exactly as she described and so then the real fun began.

I looked at the xorg.conf file from /etc/X11/ – it looked normal (it was the default config file) I first tried doing some edits to clearly specify the trackpad device (cursor movement with the trackpad had been working though, it was just the clicks that weren’t registering.

So, after making changes I restarted gdm (the greeter) ( /etc/init.d/gdm restart ) and gave a try. Some things started working, tapping the trackpad substituted for the left click and the right click work. The external mouse worked with both buttons.

So, I chipped away a bit more. I verified that the files existed for some of the appropriate packages, some I reinstalled. I uninstalled compiz (having read a bug report related to that) I reverted to an older version of xinput-synaptic and came back to current. I tried mapping the mouse buttons with an .xmodmap file.

Nothing productive.

One thing that I did notice was that the right trackpad or both mouse buttons would work only when I had restarted gdm, not on a freshboot. So, I started looking at the boot process and the output in /var/log/Xorg.0.log to see if there was anything significantly different being reported there. Nothing useful.

I disabled the “dontzap” feature that’s new in jaunty. If you read the release notes they have disabled the ability to “kill off” the graphical interface with ctl-alt-backspace… I essentially re-enabled that option so it’s quickly and easily possible to restart the gui.

The bottom line of what I found was this. If you’re moving the mouse (trackpad) when gdm first starts, the mouse buttons will be usable (with the exception of the left button on the trackpad (!)) If you don’t move the mouse at the startup of gdm it get’s “stuck” in pointer mode… I can bring the cursor down over the “options” menu in the lower left corner of the greeter screen and it still holds it’s i-beam icon as though you are editing a text window, the options does not highlight on the mouseover and no right/left click/tapping will work until gdm is blown away and restarts with a ctl-alt-backspace.

After that everything works with the exception of the left trackpad button (although you can tap.) I tried reversing the buttons and it’s still the one on the left that doesn’t function.

Wondering if I was missing something I tried an experiment. I (using another computer) installed Jaunty 9.04 to a USB hard drive and brought it up to date with all the current updates. I booted this usb hard drive on three or four systems and it worked marvelously (hardware detection “just worked” on each machine.) The only one to exhibit any problems was this Acer Aspire One ZG5. Guess what the problems were? well…. if I didn’t move the mouse when the greeter screen first came up I wasn’t able to do any mouse clicking (the pointer seemed frozen in the i-beam cursor icon mode) and even if I did (or restarted gdm) the left button did not work.

I came to the conclusion that there were two different issues, one I suspected may be a software issue and an update could fix it (the initial boot problem with useless buttons.) The second problem (left click not working) could just be a hardware issue – since it only affects the trackpad and not an external mouse. As I think about it, I do wonder if the trackpad were faulty (left button) could it cause the other problem when it’s first initialized (?)

If I knew I would have access to the machine again I might have filed a bug report with Ubuntu to see if they could get someone to verify this as software issue (or not). SInce I won’t have access to it, I wouldn’t be able to follow up with more information if they needed it, so instead of a bug report I’m posting here.

The only way to know for sure if it’s Ubuntu 9.04 or the netbook is to have an identical model to test, which I don’t.

So, does anyone out there have the same IDENTICAL model netbook – with jaunty 9.04 – are you seeing a similar problem or does it work just fine? The updates on this machine were current as of June 1, 2009.

Comments are open in case anyone has some input on the issue.

So, I’ve spent some time recently working with Zarafa. It’s an exchange replacement that has a completely open source version as well as a version with licenses for outlook if you want that level of integration. It’s web interface looks good and is an improvement on Exchange server’s capabilities with a non-IE browser. Anyway it can install on a linux server and integrate with several different MTA’s. Our choice was postfix since that was already installed for the purposes of processing forms on a webserver on the same host. Outgoing mail “just worked”, but postfix needed a bit of configuration to make the incoming mail work.

Our system is Ubuntu 8.04 and the original postfix setup had been a system that only sends mail and doesn’t receive. SO, instead of tinkering with the configuration files I moved them out of the way (backed them up as our working config actually.) Then I ran dpkg-reconfigure postfix to change our operating profile to that of an internet station (meaning that we would be sending and receiving mail directly.) This server is a VPS server hosted at slicehost. Anyway, the trick was configuring postfix for delivery to zarafa. I found a few conflicting suggestions, but finally settled on the following line being added to main.cf

mailbox_transport = zarafa: zarafa_destination_recipient_limit = 1

Of course, you also need to specify your domain and which domains you are receiving mail for. In this case we were going to be receiving mail for multiple domains and the following suggested addition to master.cf wasn’t going to work for us.

zarafa unix – n n – 10 pipe flags= user=vmail argv=/usr/bin/zarafa-dagent ${user}

You see, our goal was two domains with the same email frontend ….. i.e. mail@domain1.com and mail@domain2.com to be seperate accounts. By default the “user” string above was going to take our address and crop the email address before the @ sign as user… so both of those accounts would look the same… they would both be the user mail.

SO, I did a bit of digging on what variables postfix takes in it’s master.cf file. I found a bit of advice to do a “man 8 pipe” since that’s the utility that makes postfix’s delivery happen. So, finally I found a good reference in the pipe manual to do what I thought would work. We could use ${recipient} the only question was if zarafa-dagent could use that…

So as root I ran zarafa-dagent without options and got some good information including….

-R Resolve the primary email address passed in , and deliver to that user. Only one of -e and -R may be specified.

Woohoo…

So, I added this line to master.cf in place of the one suggested above…

zarafa unix – n n – 10 pipe
flags= user=vmail argv=/usr/bin/zarafa-dagent -R ${recipient}

Tested and all seems to be working well.

So, now we have abstraction of usernames and email addresses. You could have a username velma and the address jinkies@mydomain.com and delivery will still work. You can also have the username scooby and address jinkies@thisotherdomain.com and they will have unique mail stores (in spite of the first part of the address being the same.)

There are certainly situations where you would want user@domain1.com and user@domain2.com to be the same, but this is a good fix for those situations where you want to manage discrete accounts for multiple domains. From what I’ve seen thus far out of zarafa I’m liking it. Our next project will likely be imap integration

To make sure you’ve go the idea that mplayer is really a swiss army knife for media files that can do most anything. I wanted to get a post JUST on this idea. Any media format that mplayer recognizes it can dump the audio to the most common and readable audio format out there, the PCM Wav file.

Here’s how it goes:

Usually this is just an intermediary step en route to another file format (mp3 for instance..)

Here’s a line I’ve used from a script of mine:

mplayer -ao pcm:file=$1_$date.wav $1.rm

so… this calls mplayer and gives the audio output driver as the pcm output driver and the filename you can see I’ve controlled with the command switch again. In this case I was converting from a .rm file to the wave format. Unfortunately the rm file was not named descriptively enough for my taste, so I’ve added a variable that tags the date to the filename as well.

So… let’s say you run converttowav myaudio

Then mplayer will take the source of myaudio.rm and give you myaudio_todaysdate.wav in the nice compatible (and easily convertable) wav format.

There is a lot of software out there that can capture realmedia online streams for a fee. I’ve found a way in linux to do the same without buying third party software.

The main motivation for this is I discovered SBS (an Australian broadcaster) provides audio news programs (1 hour length) in a variety of different languages. And if you didn’t know studying languages is yet another hobby of mine.

The problem is their streams are realmedia and meant to be listened to online. I’d rather listen to them using my portable mp3 player. I’ve talked about converting rm to mp3 in a previous post… now let’s look at capturing the rm stream.

Here’s all it takes to capture that realmedia file….

mplayer -dumpstream -playlist http://addressto/ramgen/record/$1.rm -dumpfile $1.rm

In this case we use mplayer to dump the stream, we give it the address that the playlist is generated from. Since I’m studying more than one language I pass the language to the script I’ve got this in as a variable so… getstream spanish would make $1 = spanish and the file get’s dumped to spanish.rm

On my (somewhat older) laptop, I’ve noticed that MPlayer sometimes gets the audio and video out of sync. The audio is moving faster than the video. I’ve noticed this in octoshape streaming as well as in playing standalone videos.

I discovered a quick fix to this is to use the d hotkey on the keyboard. This toggles between frame dropping modes. By default mplayer tries to draw every frame it receives, but toggling to either framedropping enabled or framedropping hard can help keep the video up with the audio track. I’ve found this has made my octoshape streams usable again! (Usually I just make use of the framedropping enabled instead of the hard setting. (I haven’t investigated the difference between the two.)

Sometime back I setup my home server running Ubuntu Linux (6.06 – Dapper Drake LTS). I used two pairs of drives to do raid cloning. Two IDE drives were for the main system structure and two SATA drives for Audio/Video storage as well as CDimages and other large file sharing on the local network. Well… I noticed the hard drive light was on solid and sure enough one of the two SATA drives had failed. (I didn’t get my status email because I’d done a network structure change and didn’t update my local mail setup…) Anyway… replacing it was a pain in the neck only for the physical access to the box. Everything else worked as it should.

I had made a file with the contents of the partition table when I first set things up which helped…

So from another pc (with the new drive attached as a usb disk) I ran…

sudo sfdisk /dev/sdb < partitiontable.sda

(sda was the one from the old setup that had failed.) Then I logged in and removed the failed drive from the arrays.

sudo su
mdadm /dev/md2 –fail /dev/sda6 –remove /dev/sda6
mdadm /dev/md1 –fail /dev/sda5 –remove /dev/sda5
mdadm /dev/md0 –fail /dev/sda1 –remove /dev/sda1

(Remember those long –’s are really a double hyphen – wordpress is funny that way and interprets them differently.)

I shut down the machine in question and carefully pulled it out where I could work on it. (Those front loading hard drive trays would be REALLY nice for my home setup.) I figured out which was sda according to the system board, disconnected it and tested the bios to make sure. Sure enough I had pulled the correct one.

I shut down again and removed the old drive entirely, replacing it with the new drive and hooking up the cables. Then I powered back up to check bios again. All is good. I powered things back down to put the cover on the case and move it back into it’s cubby hole.

All booted up just fine – doing a “cat /proc/mdstat” showed just one drive for the raid drives of md0, md1 and md2. So, I just did the following…

mdadm /dev/md2 –add /dev/sda6
mdadm /dev/md1 –add /dev/sda5
mdadm /dev/md0 –add /dev/sda1

and checked “cat /proc/mdstat” again to find that they were in process of syncing and all looked healthy. The syncing process will take a while (these are 400GB drives) especially if there’s is change to the content during this proces, but once that’s done it should be a perfectly healthy raid array again.

While I was at it I ordered a pair of drives to start rebuilding my desktop system with a raid array for the primary system file structure and the home partitions. I like the redundancy that simple software raid gives. (I’ve talked before about even adding a third drive in temporarily as a backup drive. USB may be a bit slow for doing that frequently, but as a snapshot it’s not a bad approach from time to time.)

Recently I had a request for help with some free pdf’s I’ve published elsewhere. I wasn’t really keen to spend a lot of time for free supporting free pdfs and really just wanted to give an alternative file format. (I suspected the pdf format may not have been one that their computer liked. So, I wanted to convert to a tif image as that’s something that should be universally usable. Some quick websearching reminded me that in linux it’s a very simple conversion. In fact, the command line program convert is all you need.

It was as easy as $convert filename.pdf filename.tif

wow…

Windows users know chkdsk, linux users know fsck… users of each MIGHT have heard of SMART. These are different ways of TESTING hard drives. Well, there’s also a utility called TestDisk that looks promising for recovering data… Here’s the clip from their site. “free data recovery software! It was primarily designed to help recover lost partitions and/or make non-booting disks bootable again when these symptoms are caused by faulty software, certain types of viruses or human error (such as accidentally deleting a Partition Table). Partition table recovery using TestDisk is really easy.” It runs under a variety of OS’s and recognizes several different disk formats.