Disc / RAID Configuration For HTPC [Archive]

View Full Version : Disc / RAID Configuration For HTPC

bLaXe

10-01-2007, 04:06

I'm in the process of putting together a HTPC - Core2 Duo / i965 based - which will run MythTV / Gentoo Linux. I'm looking at putting in a large (around 1TB) amount of disc space, but I'm not sure of the way to go about it.

Obviously I will be using multiple (SATA-II) discs, so RAID would seem appropriate, but what kind? I will be using the software RAID built in to the Linux kernel, which (to my knowledge at least) supports RAID levels 0, 1 and 5.

RAID-5 would be my first choice, as it gives redundancy without losing too much of the raw disc space. However, I worry about the performance of RAID-5, especially in software. There will be 2 DVB tuners initially, and I expect to be adding more in the future. I'd rather not run into a 'disc I/O' brick wall 6 months down the line when I come to upgrade.

RAID-0 would solve any worry about performance, but I've been burned in the past when one disc out of array failed (causing all the data to be lost).

I'm put off RAID-1 as I don't like losing half the raw disc space - but I might consider a RAID-1+0.

One solution I came up with was to have a small (say, 100-200GB) RAID-0 array used to hold the TV buffer and other frequently accessed/changed data, then have a much large (~1TB) RAID-5 array to hold archived media/files. This seems to me to be the idea solution, the 'best of both worlds'. It does however require a lot of drives, at least 5.

Am I over analysing this?

I'd love to hear your comments.

Tami

10-01-2007, 10:20

Raid 0 and backup your files.

Danny Boy

10-01-2007, 10:46

It all depends which data you want to protect the most. Personally, I'd go for raid 0 for my saved programs as I don't care all that much if in two years time I lose something I recorded last week. I think you may be overestimating the performance you need though. Streaming media to a hard disk isn't the most demanding of tasks. The performance of a single drive will be plenty for most users.

You also need to consider how much space you're willing to give up for hard drives. If you want 2 seperate arrays then you'll need at least 4 disks. If you're looking for a small HTPC then you'll want to stick to 1 array at most. You might also want to look at whether or not your motherboard supports hardware raid 5. Most modern motherboards with SATA controllers do support several different raid levels, and if you can implement hardware raid 5 then you'll have little to no performance difference betwen that and raid 0, but you'll get the extra space you're after.

shifty.ricky

10-01-2007, 10:54

To be fair RAID5 is fast enough. I would however use hardware RAID controller - thats just becuase I dislike software raid. To be fair DVB tuners AFAIK are a max of 1Mb a second (mostly 30Mb/minute). Now given that normal HDD can achieve 70Mb/second you're not going to bottleneck it.

Better still get one that support OCE and you are laughing. 3 x 750Gb drives gives a nice size in RAID5.

bLaXe

10-01-2007, 22:08

It all depends which data you want to protect the most. Personally, I'd go for raid 0 for my saved programs as I don't care all that much if in two years time I lose something I recorded last week.

But if your array fails you lose everything. I could handle losing my TV buffer, but my entire media archive? Not a chance - I got burned bad by RAID-0 a few years back, drive failed losing me 120GB of data. Of course, no array is 100% safe, but with RAID-0 you increase the chance of a disc failure costing you your data since any drive failing means the loss of the whole array.

I think you may be overestimating the performance you need though. Streaming media to a hard disk isn't the most demanding of tasks. The performance of a single drive will be plenty for most users.

I probably am overestimating. But like I said, I want room to grow (more tuners, remote frontends).

You also need to consider how much space you're willing to give up for hard drives. If you want 2 seperate arrays then you'll need at least 4 disks. If you're looking for a small HTPC then you'll want to stick to 1 array at most.

Well, I've just bought a Silverstone LC20M (http://www.kustompcs.co.uk/acatalog/info_1283.html) which has 6 internal 3.5" bays. So I guess you could say I'm not looking for a small HTPC :p

You might also want to look at whether or not your motherboard supports hardware raid 5.Most modern motherboards with SATA controllers do support several different raid levels, and if you can implement hardware raid 5 then you'll have little to no performance difference betwen that and raid 0, but you'll get the extra space you're after.
I may be wrong, but aren't most of the RAID controllers on motherboards not really hardware? They still require drivers (and I doubt there are Linux ones available).
Anyway - I don't like using hardware RAID if it's on the motherboard. What happens when I change the motherboard?

To be fair RAID5 is fast enough. I would however use hardware RAID controller - thats just becuase I dislike software raid. To be fair DVB tuners AFAIK are a max of 1Mb a second (mostly 30Mb/minute). Now given that normal HDD can achieve 70Mb/second you're not going to bottleneck it.

Individually, no, but like I said, I want room to grow.

Better still get one that support OCE and you are laughing. 3 x 750Gb drives gives a nice size in RAID5.

I'd love a proper hardware RAID card, but I can't really afford one - and that really would be overkill! I'm going to be running LVM2 on top of the arrays so I'll be able to resize the partitions without any trouble.

Thanks for the replies guys.

I'm still not entirely sure what I'm going to do, although I'm swinging towards a single RAID-5...

stdRaichu

12-01-2007, 04:14

I seem to have pretty much the same setup as you as far as the main Myth backend goes...! I'm well impressed with the G965 chipset, the onboard GFX work surprisingly well in Linux (yeah, even Beryl) as long as you're not planning on playing any serious games. Get one if you can stand not being able to run games on it.

I've recently splashed out on more 750GB drives (1TB drives appearing in a month or two, w00t!) and have relegated my current file server (in sig) to the role of backup server; the Myth backend uses no RAID at all, but the backup server powers itself up twice a week and, through a hideous collection of scripts (and boy, have I really fallen in love with rsync), backs up all the important stuff from all of my computers, checks them for integrity, mails me the log of what's been backed up, and then powers down again. The Myth backend also images itself (via dd) to an identical Raptor, so in the event of the Raptor dying I just replace it with a spare, reboot and I'm back off (almost) where I started. The backup server retains the superb 3ware card, although at the moment I'm using the discs in JBOD since I don't yet have enough raw disc space to put them in RAID. All of these numerous drives usually live in one of IcyBox'sCremax's reasonably priced SATA caddies.

As for RAID configuration for your home, if I were in your situation I would definitely go for RAID10 (not the same as 0+1!) - the redundancy of RAID1 and the speed of RAID0 if you can afford four large discs. Alternatively making a RAID1 array (ultimate simplicity) and then using LVM to add further RAID1 arrays later as you buy more discs.

Software RAID should be fine for your purposes, although personally I would stay away from RAID5 - the thing I hate most about recovering from RAID failure (be it software RAID5 or hardware RAID) is that you can't just yank a disc, stick it in another box and chroot into it - 3ware discs will only work on a 3ware controller, and with softraid5 (or indeed any other array that uses stripes or parity) needs to be rebuilt. Since no home user ever keeps a hot spare, I would prefer to go for the most easily recoverable array, which IMHO would be with RAID1 + LVM option.

Net result of all of this doohickery is that a) I now have a proper backup regime that means I can recover from HDD failures and accidental deletions damned quickly, I've managed to turn off an entire computer (for most of the time), using less energy and thereby allowing people other than myself to actually sleep in my room, although I am finding I miss the white noise of the fans in that 4U :D

As to your question - are you overanalysing this? Yes, and so are the rest of us :P

And yeah, the "RAID" you get on motherboards isn't RAID at all - all the RAID work is done in the ATA controller driver, and the controller itself is only used as a dumb pipe for data from the RAID driver. Proper hardware RAID cards typically include their own processor, CPU and RAM, usually with an embedded command line utility or web application as well as utilities to recovery from almost every possible sort of array failure, and they're a completely different beast to the piddling pseudo-RAID you get on mobos. They're also often £300+ for a card with 8 or more SATA ports. Very nice for those of us who can afford it, a great excuse to scour eBay for those that can't :D

Karl

12-01-2007, 12:47

You might also want to look at whether or not your motherboard supports hardware raid 5. Most modern motherboards with SATA controllers do support several different raid levels, and if you can implement hardware raid 5 then you'll have little to no performance difference betwen that and raid 0, but you'll get the extra space you're after.

I don't think I've come across a motherboard yet that has true hardware RAID-5 support on it. The onboard RAID is usually just a controller for masking the fact that a "drive" in the OS is actually several drives, they don't usually do much other then that, and handle the drive writes - They usually make the OS do all the computation - Which for RAID5 can be very expensive in terms of CPU. True hardware controllers will have hardware XOR engines on them along with lots of other circuitry - That's why they cost £££.

bLaXe

12-01-2007, 18:40

@stdRaichu

Looking at your sig, we do seem to be on the same page. Gentoo is just too good I suppose :rolleyes: . I notice you're running x86_64, as I plan to. Any pitfalls with 64bit MythTV I should know about?

I'm in process of rebuilding my desktop/workstation to use RAID-1/LVM (2x120GB + 2x80GB). Since playing with LVM on a couple of servers I just had to have it at home. I do nightly backups with rdiff-backup from all my boxes to a USB hard disc (permanently) attached to my desktop. IIRC rdiff-backup uses rsync under the covers. Rsync is generally pretty cool though. Life without rsync/ssh (and screen for that matter) would be so much harder.

At some point though I plan to build a NAS box which could handle backups, among other things. I've been dying for an excuse to play with *BSD (shame about Gentoo/FreeBSD being on hold because of that license thing), and putting together a NAS seems like a good excuse. Ideally that would involve a proper RAID controller card. Of course, this relies on my having any money...

Back on topic, I'm still erring towards RAID-5...

RAID-10 would, as you (and some other people I've spoken to) say, be the best of both worlds, but it's a bit out of my price range. To get the magic 1TB I'm looking at a minimum of about £400, whereas with RAID-5 it's closer to £300 (plus I'd only be using half of the available 3.5" bays). It's the space efficiency really, 50% for RAID-10, 66% for RAID-5. I'm a cheapskate at heart.

I take the comments about annoying rebuilds with RAID-5, but should the worst happen I am generally just happy I still have my data - a few hours spent rebuilding arrays is fine by me - any excuse for a couple of pints.

I like the idea of adding more arrays further down the line - in fact this is one of the main reasons for using LVM in the first place. I plan to do this with RAID-5 arrays. There onboard controller has 6x SATA ports, and the case has 6x 3.5" bays - a match made in heaven. A 3 disc array to start with would allow me to add another, larger array in the future.

I'm still not certain what I'm going to do - I''ll sleep on it for a day or two.

I received the rest of the parts (minus hard discs) for the HTPC this morning. I had planned to experiment with one of the many unused PATA discs I have knocking about. I had some problems getting the Gentoo installcd to boot. The disc controller (jmicron something-or-other) isn't properly supported, so you have to use the 'all-generic-ide' boot option (which causes rubbish I/O performance) the boot the CD. Ok, fine - it works. Kind of.... Grub refuses to play nice - complaining that the "device has no BIOS drive" when running 'grub-install'. Grrr... There's a couple of other things I have yet to try, but work got in the way this afternoon. Should have some time to work on it over the weekend though.

I never expected a smooth ride getting this box up and running - it's no fun if it all just works.

stdRaichu

13-01-2007, 17:43

Not really any pitfalls with AMD64 MythTV - like pretty much all other software, it works perfectly. Where 64bit Linux falls down on the desktop uis annoying things like Flash or the 32bit binary win32codecs, all of which neccesitate running a 32bit environment simultaneously, which eats up shedloads of RAM. Obviously if the machine is to be used just as a Myth frontend/backend then you shouldn't have any need for any 32bit executables, and most apps now get a nice performance boost out of all those extra registers - including MySQL and ffmpeg wcich shoud be good news for all MythTVers.

If I can tempt you with anything resembling a 3ware card, I imagine you'd like it - they support dynamic expansion of the arrays as you add more discs to them, so it's like an LVM in the sense you just add the discs, tell the RAID controller to incorporate them into one of the RAID arrays, and then run extend2fs or whatever on the expanded partition, it's very cool.

I also had horriffic problems with that damned JMicron controller as well - luckily I had an existing install (compiled for AMD64 but it runs on the Core2 just fine) so I just rsynced that over to the new disc, chrooted into it and built a 2.6.19 kernel (which has full support for the JMicron as well as all the funky new libata stuff) and installed a boot block. Just remember that Linux treats the JMicron as a SCSI adapter if you use libata, so you'll need to build SCSI CDROM support, and the device node'll be something like /dev/sr0 instead of /dev/hdc.

Beware of using IDE in software RAID - if two drives are on the same channel, they can't both read and write at the same time, giving you awful performance - a problem I experienced for myself when I set up my first RAID array. SATA doesn't have these problems of course.

P.S. - just noticed from your sig that you're going to be using a Nova-T 500 - be warned that the Linux driver isn't all there yet and some users are observing problems, the most famous of which is the USB disconnect. Sine the board itself is two USB DVB chips joined onto a PCB and the Linux DVB stack didn't properly support this particular setup until recently, the drivers are still a bit green - keep a Nova-T hanging around just in case! There's no support for the remote yet either. Keep an eye on the linuxtv mailing list about it.

bLaXe

14-01-2007, 17:17

Firstly, many thanks for the comments/info - I appreciate it.

If I can tempt you with anything resembling a 3ware card, I imagine you'd like it - they support dynamic expansion of the arrays as you add more discs to them, so it's like an LVM in the sense you just add the discs, tell the RAID controller to incorporate them into one of the RAID arrays, and then run extend2fs or whatever on the expanded partition, it's very cool.

I'd love a nice 3ware card, but I think I've dented the bank balance enough for the time being. It would probably be overkill anyway - I only live with one other person, so there's never realistically going to be more than 2 frontends. Unless the "run UTP cable out of window over the road to mates house" plan ever materialises that is...

A thought occurs - can you get PCI-E 3ware cards? I only have 3 PCI slots, but there are 3 PCI-E slots as well (4x, iirc). In fact, can you get much PCI-E hardware at all (besides gfx cards, of course)? Just a thought - not actually planning to buy anything...

I also had horriffic problems with that damned JMicron controller as well - luckily I had an existing install (compiled for AMD64 but it runs on the Core2 just fine) so I just rsynced that over to the new disc, chrooted into it and built a 2.6.19 kernel (which has full support for the JMicron as well as all the funky new libata stuff) and installed a boot block. Just remember that Linux treats the JMicron as a SCSI adapter if you use libata, so you'll need to build SCSI CDROM support, and the device node'll be something like /dev/sr0 instead of /dev/hdc.

Hmm - I wonder if this might be my problem. Should IDE discs appears as SCSI devices when connected to the JMicron? I had been addressing the IDE disc as '/dev/hdk' - which grub refuses to install on. This warrants some investigation. Looks like I've found something to do this afternoon.

Beware of using IDE in software RAID - if two drives are on the same channel, they can't both read and write at the same time, giving you awful performance - a problem I experienced for myself when I set up my first RAID array. SATA doesn't have these problems of course.

Well yeah - surely that's a given? In my desktop each of the four drives in the RAID has it's own IDE channel. Two on the motherboard, two on an old cheapo Adaptec IDE RAID card which I am using simply as an IDE controller. My first RAID was a RAID-0 I used when I was still doing studio/recording work. Got burned bad when one of the drives died :mad: I was not a happy man, and now have reservations about RAID-0.

P.S. - just noticed from your sig that you're going to be using a Nova-T 500 - be warned that the Linux driver isn't all there yet and some users are observing problems, the most famous of which is the USB disconnect. Sine the board itself is two USB DVB chips joined onto a PCB and the Linux DVB stack didn't properly support this particular setup until recently, the drivers are still a bit green - keep a Nova-T hanging around just in case! There's no support for the remote yet either. Keep an eye on the linuxtv mailing list about it.

I'm aware of the state of support the Nova-T 500 - but I'm prepared to give it a go. I thought about getting a pair of Nova-Ts instead, but in the end I decided to save a PCI slot and go with a dual tuner. Like I said before - I like a challenge - I just hope I'll be able to get it working satisfactorily.

I notice that you have a Nova-T 500 in one of your Myth boxes, so I might have to pick your brains sometime if I get really stuck :p

stdRaichu

18-01-2007, 03:14

Firstly, many thanks for the comments/info - I appreciate it.

No problem :D

I'd love a nice 3ware card, but I think I've dented the bank balance enough for the time being. It would probably be overkill anyway - I only live with one other person, so there's never realistically going to be more than 2 frontends. Unless the "run UTP cable out of window over the road to mates house" plan ever materialises that is...

Yeah, I guess they are a bit overkill for a home environment...! But fi balls-to-the-wall performance is important they're a fantastic buy - my workstation utterly flew with two 74 raptors in RAID1 (cos the 3ware is clever enough to stripe the reads, so you get the resiliency of RAID1 and the read speed of RAID0).

A thought occurs - can you get PCI-E 3ware cards? I only have 3 PCI slots, but there are 3 PCI-E slots as well (4x, iirc). In fact, can you get much PCI-E hardware at all (besides gfx cards, of course)? Just a thought - not actually planning to buy anything...

I believe 3ware have just brought out a range of PCIe RAID cards. Areca and LSI have also had them about for a while too. IIRC they either use 4x or 8x lanes, but of course technically should run in a 1x slot.

Hmm - I wonder if this might be my problem. Should IDE discs appears as SCSI devices when connected to the JMicron? I had been addressing the IDE disc as '/dev/hdk' - which grub refuses to install on. This warrants some investigation. Looks like I've found something to do this afternoon.

Yup, the JMicron driver has only made it into libata so far, which is kinda half-PATA, half SCSI (since the SCSI stack already had the code for things like hot-swap and command queueing). HDD's in it should be /dev/sdX and optical drives from the PATA controller will be /dev/srX. udev should still handle the human-readable device nodes a la /dev/cdrom for you.

You have to pay attention to the BIOS settings too though - there's usally an option in the BIOS as to what "mode" you want your JMicron running in - FWIW I'm using "IDE and AHCI".

Well yeah - surely that's a given? In my desktop each of the four drives in the RAID has it's own IDE channel. Two on the motherboard, two on an old cheapo Adaptec IDE RAID card which I am using simply as an IDE controller. My first RAID was a RAID-0 I used when I was still doing studio/recording work. Got burned bad when one of the drives died :mad: I was not a happy man, and now have reservations about RAID-0.

Heh, same here...! Although I never actually lost anything, I've had to explain to alot of people how their 733t RAID0 has died and taken all of their never-backed-up data with it.

I'm aware of the state of support the Nova-T 500 - but I'm prepared to give it a go. I thought about getting a pair of Nova-Ts instead, but in the end I decided to save a PCI slot and go with a dual tuner. Like I said before - I like a challenge - I just hope I'll be able to get it working satisfactorily.

I notice that you have a Nova-T 500 in one of your Myth boxes, so I might have to pick your brains sometime if I get really stuck :p

It's not actually running under Myth at the moment - those USB disconnects usually cause myth to segfault (grrr) if they happen when it's recording stuff. At the moment I just use them in Kaffeine when t'other recorders (two Nova-T's and an Avermedia 771) are busy. Be warned that at the moment using it is... er, well, remarkably simple really - you just need kernel support for the DibCom 3000 frontends.

Hope that's of some use for your demi-god box :D

stdRaichu

18-01-2007, 03:37

As an aside, I've finally managed to convince Myth to pick up the BBC HD broadcast. My supposedly non-HD capable Nova-T's (kids, don't be fooled by all this crap about needing an HD-compatible TV card - it's all ******** to get you to part with your cash. All a DVB card does is dump a bunch of bits onto a disc, it doesn't matter what the broadcast is) capture the stream just fine'n'dandy, the only problem is I can't find and H264 decoder that'll work yet - apparently it's a known problem so here's fingers crossed... the really crazy thing is that Myth has managed to create a thumbnail of it! Weird...

And yeah, it's chomping up disc space like you wouldn't believe! But Alan Pitchfork's voice has never sounded so sweet :D

bLaXe

19-01-2007, 19:34

stdRaichu

24-01-2007, 01:13

I've finally got around to ordering the discs: 3* 500GB SATA-300 (Seagate 7200.9, 16MB 8.5ms). Should arrive on Tuesday :)

I've been thinking about how to partition up the space. I have a lot (300GB or so) of media which I will be copying to the HTPC once it's up and running. It seems sensible to me to keep the MythTV recordings separate from the media archive (not sure exactly why, it just seems sensible). So, the bulk of the space will be split between two partitions.

The great thing about *nix's extremely flexible partitioning system is that you can tune each filesystem to the sort of data that's going to be stored on it. To use your example above, there's a big difference in filesystem layout between your average /usr/bin folder and your MythTV store - files on the myth store will typically be at least 500MB in size, usually far bigger (e.g. my recording of One Upon a Time in America t'other night hit 8.5GB and the HD streams from BBC are colossal - 4.5GB for 28mins!). So you tune the filesystem to be good at dealing with large files - either use something like JFS or XFS (both of which are the best at handing large files), or use ext3 with the largefile4 option (one inode [or "cluster" in windoze-speke] per 4MB meanign that you have very few inodes and each file takes up at least 4MB) and keep "standard" filesystem options for your system partitions. Once you delve into it, there's a whole shedload of optimisation you can make - granted, it usually makes negligible difference but for some tasks it can save you a world of pain.

If you want the ultimate in flexibility, you'll use LVM obviously.

Since I'm using LVM I'm only going to allocate some of the space initially - I'm not sure how much space I will need for TV etc.. I'm thinking of using 400GB for the media archive 200GB for TV. Lots of room to grow :)

Depends on how much of a hoarder you are :D I have a whole 750GB drive set up for myth, and I still find it too small. Guess I need to find a better way of archiving my TV to MPEG4...! I'd say keep the large stuff all on the same partition though - nothign is worth than needing more space in a hurry and then having to go through the hassle of resizing the filesystems (urgh).

The problem I have is booting the Gentoo installcd. No JMicron driver in the installcd kernel. To get around this I have to boot the CD with the 'all-generic-ide' option. I have a feeling this screws up the device names, as well as confusing grub-install.

One solution I have seen is to use a 3rd party installcd, which does include the JMicron driver. I tried this, but so far I have been unable to get any of them to boot, despite the fact they claim to support the controller.

Can't remember which boot CD I tried out - it definitely worked, cos I used it to do a boot block, but I have no idea what (if any) special options I needed to boot it with... sorry 'bout that! Pretty sure I never used the all-generic-ide option though; not exactly sure what it does either. All I'll say is that make sure your BIOS reports the JMicron to be in IDE+AHCI mode, at leats if my mobo is anything to go by - I had a few problems getting the boot CD to work until I changed it. Any live CD with 2.6.19 or higher should support the chip without any special options. In fact here's a post from Alan Cox (http://lkml.org/lkml/2006/7/12/133) about his patch for it. What kernels have your boot CD's been using?

bLaXe

29-01-2007, 08:27

Well, I got my three 500GB discs on Tuesday, and I'm slowly beginning to put the system together.

In the end I had no trouble with the install. The Intel ICH SATA controller works nicely, even when the installcd is booted with 'all-generic-ide'. The problems I had before seem the be caused from attempting to boot from the JMicron PATA controller and since I never plan on attempting this again I'm just going to leave it.

I went with a big RAID-5 array in the end, with a RAID-1 for /boot, RAID-0 for swap and a little RAID-5 for / (since I don't know a way of having / on LVM). After all said and done, I'm left with an LVM2 volume group of 928MB, which is pretty tasty. I've used XFS for the media/mythtv partitions (with block size=4096), and reiserfs for the system partitions (since I know from experience it can both grow and shrink file systems without issues).

I'm running the stock Gentoo 2.6.19 kernel (still ~arch masked, but I know I'll eventually need >=2.6.19 to get Nova-T 500 support). For now though I'm using a single Nova-T PCI (I have another, but I haven't bought an aerial splitter yet!). I've had a go at getting the tuner card to work, but as yet I've been unsuccessful. There's quite a bit of information around, but I can't seem to find anywhere which tells me exactly which kernel options I need. Some more googling is needed... I haven't got as far as installing myth itself, since I want to get the tuner card working first.

I am planning to transcode the mythtv recordings into something else for archive purposes. At the moment I'm planning to use theora/ogg (love the free codecs), but I'll have to see what's the easiest way to get video out of myth.

It's a bit of a struggle finding the time to work on the box at the moment though. Damn the need to work (well, kind of) for a living :mad:

On a side note, and without sounding too much like a fanboy... OMFG!!! Core2 Duos are soooo fast!

stdRaichu

30-01-2007, 01:36

Crikey, we do seem to love making such long posts. Anyway, I'm going to skip the bits of your post that are too convoluted to go into past my bedtime and crack on with...

I'm running the stock Gentoo 2.6.19 kernel (still ~arch masked, but I know I'll eventually need >=2.6.19 to get Nova-T 500 support). For now though I'm using a single Nova-T PCI (I have another, but I haven't bought an aerial splitter yet!).

Which rev Nova-T are you using? I use the 909 revs, and they handily come with a loopback connector for running series of them together, therefore obviating the need for splitters. Snip from lspci:
04:00.0 Multimedia video controller: Brooktree Corporation Bt878 Video Capture (rev 11)
04:00.1 Multimedia controller: Brooktree Corporation Bt878 Audio Capture (rev 11)
04:01.0 Multimedia video controller: Conexant CX23880/1/2/3 PCI Video and Audio Decoder (rev 05)
04:01.2 Multimedia controller: Conexant CX23880/1/2/3 PCI Video and Audio Decoder [MPEG Port] (rev 05)
04:01.4 Multimedia controller: Conexant CX23880/1/2/3 PCI Video and Audio Decoder [IR Port] (rev 05)

I've had a go at getting the tuner card to work, but as yet I've been unsuccessful. There's quite a bit of information around, but I can't seem to find anywhere which tells me exactly which kernel options I need. Some more googling is needed... I haven't got as far as installing myth itself, since I want to get the tuner card working first.

No problem - here's a snip of my .config. It's from the current CK patchset (which I heartily recommend as the best desktop kernel around) but should be exactly the same on gentoo and vanilla Linux:

#
# Multimedia devices
#
CONFIG_VIDEO_DEV=m
CONFIG_VIDEO_V4L1=y
CONFIG_VIDEO_V4L1_COMPAT=y
CONFIG_VIDEO_V4L2=y

#
# Video Capture Adapters
#

#
# Video Capture Adapters
#
# CONFIG_VIDEO_ADV_DEBUG is not set
CONFIG_VIDEO_HELPER_CHIPS_AUTO=y
CONFIG_VIDEO_TVAUDIO=m
CONFIG_VIDEO_TDA7432=m
CONFIG_VIDEO_TDA9875=m
CONFIG_VIDEO_MSP3400=m
CONFIG_VIDEO_CX2341X=m
# CONFIG_VIDEO_VIVI is not set
CONFIG_VIDEO_BT848=m
CONFIG_VIDEO_BT848_DVB=y
CONFIG_VIDEO_SAA6588=m
# CONFIG_VIDEO_PMS is not set
# CONFIG_VIDEO_CPIA is not set
# CONFIG_VIDEO_CPIA2 is not set
# CONFIG_VIDEO_SAA5246A is not set
# CONFIG_VIDEO_SAA5249 is not set
# CONFIG_TUNER_3036 is not set
# CONFIG_VIDEO_STRADIS is not set
# CONFIG_VIDEO_ZORAN is not set
# CONFIG_VIDEO_SAA7134 is not set
# CONFIG_VIDEO_MXB is not set
# CONFIG_VIDEO_DPC is not set
# CONFIG_VIDEO_HEXIUM_ORION is not set
# CONFIG_VIDEO_HEXIUM_GEMINI is not set
CONFIG_VIDEO_CX88=m
# CONFIG_VIDEO_CX88_ALSA is not set
CONFIG_VIDEO_CX88_BLACKBIRD=m
CONFIG_VIDEO_CX88_DVB=m
CONFIG_VIDEO_CX88_VP3054=m

#
# V4L USB devices
#
# CONFIG_VIDEO_PVRUSB2 is not set
# CONFIG_VIDEO_EM28XX is not set
# CONFIG_USB_VICAM is not set
# CONFIG_USB_IBMCAM is not set
# CONFIG_USB_KONICAWC is not set
# CONFIG_USB_QUICKCAM_MESSENGER is not set
# CONFIG_USB_ET61X251 is not set
# CONFIG_VIDEO_OVCAMCHIP is not set
# CONFIG_USB_W9968CF is not set
# CONFIG_USB_OV511 is not set
# CONFIG_USB_SE401 is not set
# CONFIG_USB_SN9C102 is not set
# CONFIG_USB_STV680 is not set
# CONFIG_USB_ZC0301 is not set
# CONFIG_USB_PWC is not set

#
# Radio Adapters
#
# CONFIG_RADIO_CADET is not set
# CONFIG_RADIO_RTRACK is not set
# CONFIG_RADIO_RTRACK2 is not set
# CONFIG_RADIO_AZTECH is not set
# CONFIG_RADIO_GEMTEK is not set
# CONFIG_RADIO_GEMTEK_PCI is not set
# CONFIG_RADIO_MAXIRADIO is not set
# CONFIG_RADIO_MAESTRO is not set
# CONFIG_RADIO_SF16FMI is not set
# CONFIG_RADIO_SF16FMR2 is not set
# CONFIG_RADIO_TERRATEC is not set
# CONFIG_RADIO_TRUST is not set
# CONFIG_RADIO_TYPHOON is not set
# CONFIG_RADIO_ZOLTRIX is not set
# CONFIG_USB_DSBR is not set

#
# Digital Video Broadcasting Devices
#
CONFIG_DVB=y
CONFIG_DVB_CORE=y
# CONFIG_DVB_CORE_ATTACH is not set

#
# Supported SAA7146 based PCI Adapters
#
# CONFIG_DVB_AV7110 is not set
# CONFIG_DVB_BUDGET is not set
# CONFIG_DVB_BUDGET_CI is not set
# CONFIG_DVB_BUDGET_AV is not set

#
# Supported USB Adapters
#
# CONFIG_DVB_USB is not set
# CONFIG_DVB_TTUSB_BUDGET is not set
# CONFIG_DVB_TTUSB_DEC is not set
# CONFIG_DVB_CINERGYT2 is not set

#
# Supported FlexCopII (B2C2) Adapters
#
# CONFIG_DVB_B2C2_FLEXCOP is not set

#
# Supported BT878 Adapters
#
CONFIG_DVB_BT8XX=m

#
# Supported Pluto2 Adapters
#
# CONFIG_DVB_PLUTO2 is not set

#
# Supported DVB Frontends
#

#
# Customise DVB Frontends
#
# CONFIG_DVB_FE_CUSTOMISE is not set

#
# DVB-S (satellite) frontends
#
CONFIG_DVB_STV0299=m
CONFIG_DVB_CX24110=m
CONFIG_DVB_CX24123=m
CONFIG_DVB_TDA8083=m
# CONFIG_DVB_MT312 is not set
CONFIG_DVB_VES1X93=m
CONFIG_DVB_S5H1420=m
CONFIG_DVB_TDA10086=m

#
# DVB-T (terrestrial) frontends
#
# CONFIG_DVB_SP8870 is not set
CONFIG_DVB_SP887X=m
CONFIG_DVB_CX22700=m
CONFIG_DVB_CX22702=m
CONFIG_DVB_L64781=m
CONFIG_DVB_TDA1004X=m
CONFIG_DVB_NXT6000=m
CONFIG_DVB_MT352=m
CONFIG_DVB_ZL10353=m
CONFIG_DVB_DIB3000MB=m
CONFIG_DVB_DIB3000MC=m

#
# DVB-C (cable) frontends
#
CONFIG_DVB_VES1820=m
CONFIG_DVB_TDA10021=m
# CONFIG_DVB_STV0297 is not set

#
# ATSC (North American/Korean Terrestrial/Cable DTV) frontends
#
CONFIG_DVB_NXT200X=m
CONFIG_DVB_OR51211=m
CONFIG_DVB_OR51132=m
# CONFIG_DVB_BCM3510 is not set
CONFIG_DVB_LGDT330X=m

#
# Tuners/PLL support
#
CONFIG_DVB_PLL=m
CONFIG_DVB_TDA826X=m
# CONFIG_DVB_TUNER_MT2060 is not set

#
# Miscellaneous devices
#
CONFIG_DVB_LNBP21=m
CONFIG_DVB_ISL6421=m
CONFIG_DVB_TUA6100=m
CONFIG_VIDEO_TUNER=m
CONFIG_VIDEO_BUF=m
CONFIG_VIDEO_BUF_DVB=m
CONFIG_VIDEO_BTCX=m
CONFIG_VIDEO_IR=m
CONFIG_VIDEO_TVEEPROM=m
# CONFIG_USB_DABUSB is not set
and the modules that I load from /etc/modules.autoload.d (not sure which are loaded by hotplug/udev by default but there you go). Note that this is also for an Avermedia 771 and it's been so long I forget which modules do with which:
# i2c
i2c-algo-bit
i2c-dev

# v4l
v4l1-compat
v4l2-common

# tv cards
cx88-dvb
dvb-bt8xx
bttv
btcx-risc
mt352
cx8800
cx8802
cx88xx
cx88-blackbird
cx22702
ir-kbd-i2c

IIRC the kernel defaults to "Enable Video For Linux API 1 compatible Layer" rather than "Enable Video For Linux API 1 (DEPRECATED)" which you need to activate the required options; that's how it worked for me anyway.

I am planning to transcode the mythtv recordings into something else for archive purposes. At the moment I'm planning to use theora/ogg (love the free codecs), but I'll have to see what's the easiest way to get video out of myth.

Easiest way for me was to let Myth do all the hard work; it's quite easy to set up a custom user job in mythtv-setup (or mythsetup, depending on version) pointing at a crummy little bash script:
prospero ~ # cat /home/mythtv/.mythtv/mpeg-ts2mpeg-ps.job
#!/bin/bash
# Custom user job designed to convert the recorded MPEG2-TS files from the DVB recorders into MPEG2-PS files for editing in AviDemux or similar

# This script needs the folowing variables in this exact order to run:
# %DIR%
# %FILE%
# %TITLE%
# %SUBTITLE%

# So the prog will be called like: /path/to/script "%DIR%" "%FILE%" "%TITLE%" "%SUBTITLE%"

echo "Starting Replex job..."

# Locate the file speified
SRC_DIR="$1/" # %DIR% plus a trailing slash
SRC_FILENAME=$2 # %FILE%

SRC_FILE="${SRC_DIR}${SRC_FILENAME}"

echo "Source file is $SRC_FILE"

# Specify output path
DEST_BASE_DIR=/home/mythtv/myth2avidemux/

# Save it to dir with name of show for easier handling of things
DEST_DIR="/home/mythtv/myth2avidemux/$3/"

# Check the destination directory exists
if [ ! -d "$3" ];
then
mkdir -p "$DEST_DIR"
fi

# Form filename
DEST_FILENAME="$3 XX-XX $4 [DVB].mpeg2"

DEST_FILE="${DEST_DIR}${DEST_FILENAME}"

# TODO: See if the destination filename exists
if [ -e "$DEST_FILE" ];
then
# TODO: if it does already exist, prepend the date/time or something
$FILE_PREFIX = `date +%y%m%d-%l%M`
$DEST_FILE="${DEST_DIR}${FILE_PREFIX}${DEST_FILENAME}"
fi

echo "Destination file is $DEST_FILE"

# Run the file through replex
replex --type MPEG2 --of "$DEST_FILE" "$SRC_FILE"

echo "Finished Replexing $DEST_FILE"

# Exit nicely
exit 0
That's it in it's current form anyway - been a while since I've had the time to do any transcoding so I'm not sure if that script still works, and it may even have glaring syntax errors in it... but the gist of this is, you can use Myth's internal %something% variables to pass things like the show name and episode title to any script you like; then just use Myth's auto-job-execution functionality to e.g. automatically archive every episode of The Simpsons. I do all my editing in an external editor a la avidemux2 since I can get frame-perfect commercial edits that way.

On a side note, and without sounding too much like a fanboy... OMFG!!! Core2 Duos are soooo fast!

Damn straight... I only have the 1.86GHz version and it's still like slurry off a shovel :D

Mon Sep 18 18:00:15 2006 >>> sys-libs/glibc-2.4-r3
merge time: 26 minutes and 31 seconds.

Thu Nov 9 12:08:06 2006 >>> sys-libs/glibc-2.4-r4
merge time: 26 minutes and 39 seconds.

Fri Nov 10 13:23:48 2006 >>> sys-libs/glibc-2.4-r4
merge time: 15 minutes and 6 seconds.

Wed Nov 15 22:07:00 2006 >>> sys-libs/glibc-2.4-r4
merge time: 18 minutes and 5 seconds.

Guess when it was I got my C2D...? :D

bLaXe

30-01-2007, 02:30

Thanks for the kernel config - I'll give it a try when I get the time to sit down and have a play.

Quickly (?) though...

Which rev Nova-T are you using? I use the 909 revs, and they handily come with a loopback connector for running series of them together, therefore obviating the need for splitters.
Snip from lspci:
04:00.0 Multimedia video controller: Brooktree Corporation Bt878 Video Capture (rev 11)
04:00.1 Multimedia controller: Brooktree Corporation Bt878 Audio Capture (rev 11)
04:01.0 Multimedia video controller: Conexant CX23880/1/2/3 PCI Video and Audio Decoder (rev 05)
04:01.2 Multimedia controller: Conexant CX23880/1/2/3 PCI Video and Audio Decoder [MPEG Port] (rev 05)
04:01.4 Multimedia controller: Conexant CX23880/1/2/3 PCI Video and Audio Decoder [IR Port] (rev 05)
My lspci output is same as yours for the Conexant devices (lspci -v reveals I do have the 909 revs). I do not have the 'Brooktree' devices. Maybe I'm spinning out, but those devices aren't even on the Nova-T PCI are they?

Did not know about the loopback connector. I'll look into it. (...wonders if you can chain the Nova-T 500 as well....)

Crikey, we do seem to love making such long posts.
I make no apologies for length or girth...

bLaXe

31-01-2007, 01:41

Just tried the kernel config - worked a treat. Many thanks!

About this loopback connector - I can't find any reference to it anywhere, and I can't see anything that looks like it might be one on the card itself. Am a missing something?

brumster

31-01-2007, 11:15

Blimey, missed this thread, hiding under the guise of an all-RAID discussion :p

The "new" Nova-Ts (and I've had mine for well over a year now) have RF in and out plugs on the back so you can loop the output of one into the input of the other without the need for a splitter. You do get signal degredation doing this, though, so I wouldn't attempt it with too many cards/VCRs/devices/etc

FWIW, late in the discussion now I know, I have 4x200Gb Maxtor SATA drives in the backend all making use of a cheapy 4-port SATA controller and linux software RAID - and an old 80Gb SATA as the main boot drive and scratchspace. It all works fine, performance is adequate and I'm not a hoarder so the 400Gb of space has never once approached full (I either delete stuff after I've watched it, or else transcode it or archive it off to a native DVD for storage). The array does 50 MB/sec according to hdparm, which 'aint as much as my hardware RAID controller in my main fileserver, but hey, let's be honest, it's good enough for a software-based solution (it's RAID0+1).

I've got various shell scripts for automated transcoding to various formats - DVD, XVID, PSP, replexing, etc. similar to stdR - and these can be kicked off via the user jobs in myth (well, four of them anyway!). Very handy ;) shout me if you need some...

bLaXe

01-02-2007, 06:10

I'm almost ready to actually install/configure MythTV itself now (need a spare few hours though..). I got XMLTV working last night (listings ahoy!), and I've managed to watch DVB through Mplayer. The awesomeness of Myth is so close I can taste it!

Blimey, missed this thread, hiding under the guise of an all-RAID discussion :p

Hmm, we did get a bit of topic didn't we... ah well....
If I have anymore problems I'll start a new thread.

The "new" Nova-Ts (and I've had mine for well over a year now) have RF in and out plugs on the back so you can loop the output of one into the input of the other without the need for a splitter. You do get signal degredation doing this, though, so I wouldn't attempt it with too many cards/VCRs/devices/etc

I'm assuming mine are "new" - I got them second hand, but they were bought within the last six months. I only have one RF plug on my cards though (aerial in). Maybe they've stopped putted them on new(er) cards. I need to split the signal to at least four devices, so I think I'll invest in a splitter/amplifier box rather than deal with rubbish quality signals.

The array does 50 MB/sec according to hdparm, which 'aint as much as my hardware RAID controller in my main fileserver, but hey, let's be honest, it's good enough for a software-based solution (it's RAID0+1).

You sparked my curiosity, so I ran hdparm on my system.
hdparm -tT /dev/md/4
/dev/md/4: <---- this is my big RAID-5 array
Timing cached reads: 4740 MB in 2.00 seconds = 2370.57 MB/sec
Timing buffered disk reads: 434 MB in 3.01 seconds = 144.38 MB/sec
Not too shabby! I'm pretty chuffed with that.

I've got various shell scripts for automated transcoding to various formats - DVD, XVID, PSP, replexing, etc. similar to stdR - and these can be kicked off via the user jobs in myth (well, four of them anyway!). Very handy ;) shout me if you need some...
Thanks for the offer - I may take you up on that sometime...

I've been reading about the MythArchive plugin (not in portage yet, grrr!) - sounds pretty funky. Archiving to 'proper' DVDs would be nice. Mind you, I'd still prefer to be able to use free codecs. I think I'll cross this bridge when I come to it though.

brumster

01-02-2007, 10:21

I need to split the signal to at least four devices, so I think I'll invest in a splitter/amplifier box rather than deal with rubbish quality signals.

Yes, I would take that as a definite. I had very poor quality running four devices (in the house in total, I mean) off the aerial until I got one of those 6ch splitter things from Maplin. Works a treat now.