ETV personal video recorder

ETV is my Personal Video Recorder (PVR). A PVR can record TV programs which you request, or according to rules you entered and program information which was grabbed from an external source (usually the internet).

All recordings are made on harddisk, which makes it possible to wind/rewind and seek at any speed, and even watch a program while it is still being recorded.

A special mode is LiveTV, where it records to a buffer and you watch the recording just a few milliseconds after it has been made. You still can switch to another station like with any normal TV, but also you can go back in time and watch the phrase you missed. Or freeze the program and answer the telephone.

The program allows you to setup recording rules like:
...any show called 'Buffy' whenever and wherever it is shown
...programs in the category 'Sport' on a specific channel between 00:00 and 06:00
...programs with 'Shakespeare' mentioned somewhere in the title or description
And naturally you can tag any program from the EPG. And you can just manually enter a scheduled recording from scratch.

ETV is networked, which means you can have a noisy big master in the garage, and a small silent frontend in both the livingroom and the bedroom. Or just run everything on a single machine.

Everything is based around controlling ETV as a fullscreen with a remote from normal viewing distance.

My target for ETV is to create a stable, usable, PVR.

Stable: It must be rock-solid whatever keys are pressed at what speed. It must run without system-administration except for grabber-script updates.

Usable: It must be usable with a normal TV/VCR remote based on the functions as labeled on that remote and/or the TV-screen. As far as possible it should mimic the behaviour of a tape-based VCR, as long as this does not detract from its improved capabilities.
The user manual should only be needed as a first introduction to the concepts behind PVRs (and ETV in particular), not for day-to-day usage.
Its concepts must be compatible with the way TV is broadcast and watched in Europe or even more specifically in The Netherlands.

PVR: Its primary function is being a very much improved VCR replacement, no effort will be made at this time to have ETV play music, show photographs, read mail or browse the web.

All existing linux PVRs I tried - before designing ETV - were centered around a US mindset of watching TV. Without sometimes knowing it themselves, authors of other PVRs started with a lot of US assumptions. But meanwhile in Europe:

• Reruns are not a way of life. Sometimes a show gets repeated, but this is usually after a year or more. Maybe some shows are repeated at night, but this is still more exception than rule.
• We don't get uniform EPG data for all channels or even all days, as data seems to be entered (on the sites we can grab with xmltv) from different sources. Meaning it is very dangerous to scan for literal identical descriptions. Word or phrase matching is much more reliable.
  And worse: episode information is usually lacking, and if present often unreliable.
• We often don't have frequency data available with the EPG. Every town uses its own frequency scheme, and those tables are not on the net, or if they are the formats differ so much it is not feasible to write grabbers. (And it probably would be easier to just share the table itself if anyone would feel the need).
• We don't use channelnumbers to select a station but use preset numbers of our own choosing.
• Shows are not boxed in 30-minute slots. Meaning the program guide must accomodate all different lengths starting at any time.
• Some stations run early a bit, and some run very late very often. This is very important to support, because the current PVRs which blindly follow the EPG (even if they allow a system-wide overrun) are useless for a typical European mix of many countries with both public and private stations.
• We usually don't have commercials which are reliable detectable by computer. The system where different commercials are inserted on the fly by many transporting cable companies (generating easy to detect breaks) doesn't occur here.
• On most stations shows are not that often interrupted by commercials, making automatic dectection rather unimportant.
• At least in The Netherlands external cable decoders are not very common. Things might change in the near future when digital TV is arriving, but at the moment I don't even know anyone who owns a decoder box or satellite receiver. So while ETV is designed with such a device in mind, there has not been written any line of code for its support.

Not all of these are implemented yet, this section only describes where ETV is going

• Language is C++, libraries are SDL, SDL_font, SDL_net, SDL_image and libavcodec from ffmpeg.
• Three-tier design: frontend (viewing), backend (recording), master (scheduling). None have to run on the same system. All systems stay alive when one of the others drop dead, and are able to re-connect without user-intervention. Think the frontend drops out while timeshifting: just reboot, and it will connect to the recorder, which still has been keeping the ringbuffer alive and being filled...
• Master and multiple frontends communicate changes in the schedule/recording tables. For instance: if one frontend changes an entry or a backend finishes a recording, another frontend user viewing the schedule list will see the change happen realtime on the screen.
• Clean and robust code. No special states needing to be dealt with: a recorder doesn't care if it is running for live-recording or scheduled recording, a viewer doesn't know if it views a timeshifted buffer, a recording in progress or a finalized recording.
• Settopbox design from the start, no X needed, no windowmanager, overscanned fullscreen only, remote control proof. And for easy data entry of rules, describing your own non-EPG recordings or big time housekeeping: just fire the frontend up on any PC with a keyboard attached.
• Scheduler can detect a recorder breaking down during a program, and it will create a linked additional recording for the remainder.
• The master supports automatic shutdown and wakeup on the next scheduled recording.
  Really needed in Europe if you calculate the expected electricity bill for a year of running a heavy backend 24/7...
• Timeshift-buffer not cleared while changing channels. The design is based around the idea of never emptying a buffer, even if it was watched yesterday before some reboots, the watched content should still be there.
  User preferences like empty on channel-change, reboot or usercommand will be optional and not enforced by the design.
• Instant recordings starting in the past (from the time-shift buffer) are possible, both automatically for a full show or as a selected part from the buffer.
  The recording by default continues using the same qualityprofile as the buffer and will end up as a two-file single-recording program. If a different quality is selected by the user the show will end up as a two-file two-recording program.
• Target is maximum quality on a low-cost silent frontend. Which means no fancy coolers or powersupplies. As I view the recordings only on traditional TVs, de-interlacing is a no-no. Lower qualities can grab 288 lines though, which is non-interlaced by definition.
• Support for recompression, even fully automatic (for instance record in huffyuv, automatic recompressing to MPEG4 multipass whenever the system is not recording, while still able to still watch the original recording while it is being re-compressed).
• Recordings can be multi-file. This removes any limits on recording size or duration. (Well, not really true. As recording time is counted in milliseconds in a signed 32 bit integer the limit is just short of 25 days for a single continous recording.)
  A multi-file recording continues in the same quality-profile, there is no time-gap between them and they are fully transparent to the user while playing and (re)winding.
• Recordings can be multi-part (or sometimes I say a program can be multi-recording, I don't have my terminology consistant yet :-).
  A multi-part recording (or multi-recording program :-) can have different quality-profiles, can have time-gaps between them, and when playing them there is a definite hickup while switching. Think DVD layer-transition.
  I'm not sure yet what to do while (re)winding: maybe have the user confirm each switch, or maybe just accept the hickups.
  For the curious: the multi-file and multi-part mechanisms are both used to copy (and possibly re-compress) a (partial) recording to a new recording whenever you record from the live tv buffer.
• Display of GUI-elements is independant of the video. Which means if you are watching a very low-res recording, the subtitles and such will still be shown in the maximum resolution.
• Automatic deletion of recordings happens in levels: from 'only superuser can delete' via 'new recording' through 'seen at least once' to 'deleted but salvagable'. The last category can still be recovered by the superuser, but if space is needed the scheduler will remove them, beginning with the oldest file.
  Besides these levels a recording created by a low-priority rule will be marked as 'new recording but deletable if space is needed'.
• Database used is MySQL 4.0, with a database schema according to the book. In the future I might switch to InnoDB tables with transaction support and journalling. Even so all program data is duplicated inside the recorded files. This makes it possible both to repopulate a crashed database, and to import the recording in another standalone ETV-system.
  Text search is handcoded and independant of MySQL features.
• Import of EPG data is via a separate SQL table. This is filled by any external grabber, a XMLTV parser for this in perl is included. The master program processes the data in the raw EPG table and updates its own EPG table with all valid data.
• Recording quality is user selected in a few levels (Max/High/Normal/Low), the codec and its parameters are hidden from the normal user.
  'Max' should ideally be defined to D1 in huffyuv or any other lossless solution available. Probably only to be used for the time-shift buffer, my 800 Mhz Athlon can keep up on a local disk, but it will be some time before a silent frontend can, and I don't think a 100Mbit NIC will manage in real life...
• Frontend runs on Windoze. Well, the scheduler does as well, but why would you ever want that as the backend does only linux...(and even that only because of a decent TV-grabber API missing on Windoze).
  Most important reasons to have the frontend working on Windows:
  • Credit where credit is due: this way I could use the Microsoft Visual C++ debugger while developing the multithreading GUI part of the frontend. Still beats DDD/GDB for me in these situations.
  • Easy to use an old non-linux compatible laptop for maintenance of rules and old recordings. I didn't want to spend money on an IR keyboard to use with the frontend, and the laptop lies around to play old graphic adventures anyway.
  • You can watch recordings on any windows-PC connected to the network. (For me that is a gadget, but a lot of people seem to watch TV on a small computerscreen.)
• No hardware codec support (like Hauppauge PVR-250/350). Not that I have anything against it designwise, but I can't see myself spending more money on a hardware codec than a decent satellite card would cost me. I want to squeeze as much video on a disk as posible, and have very strong opinions on the quality of the typical cable-company analog TV. No need to store that in 4Gb/hour Mpeg2 streams on disk...
  In the future ETV will probably support DVB digital reception (European standards naturally), that is a different game... In fact someone is already wording on the beginnings of DVB terrestial recepetion for ETV.
• There will be a commandline util to convert from ETVs internal format to a known file format, as a service to those wanting to hack their own VCD or DVD. ETV will probably not support those options on the frontend.

Not all of these are implemented yet, this section only describes where ETV is going

• Clean consistant interface (according to my personal preferences), behaves like a VCR in most ways if you use the remote.
• Forward/backward winding from single-step to any speed you want (not really truth, I artificially limited it to 2048 times because my brain was the limiting faktor), the more often you press the button the faster it goes. Feels to me like all the benefits from analog VCR and digital video combined.
  It also is the best usable interface for skipping commercials on the multi-origin channels I watch. Over here usually no automatic detection works 100%: no blanks, no markers, no logo's, any length commercial or leader possible.
• Menu based, expect for VCR-compatible buttons. I don't expect users to learn what PVR-special buttons on their remote do, it's just play/record/stop/pause/forward/backward and station-up/down If the remote has left/right/up/down that will be usable for navigating around a form, but forward/backward/station-up/down will suffice as well.
• A very basic telephone-like way of entering alpha-characters will be available, but for any serious description-editing a PC-keyboard should be used.
• Menus buttons are usable in different ways. You can:
  ...use cursorkeys, go to the button and press Play or OK/Enter.
  ...use numberkeys if you are not in an editing field.
  ...use the FLOF keys for all menus.
  
  For those not aquinted with Teletext and/or Philips: FLOF is a teletext term, used by me to indicate the related set of four colored keys originally used for teletext, but adapted by television builders to quickaccess their own menu options. The colors are red, green, yellow and blue, they are available on nearly all remotes over here, are always used for menu options, and usually not assigned any default functionality).
• Stations are selectable by a user defined number, frequencies can be entered as grid-channel, frequency, or by searching up/down (both grid and finetuning).
• Recording times are based on the EPG data, with a default pre-run and post-run offset based on the station which is transmitting. Recording times are set using time-entry or offset from the original show. If the show is from 16:00 to 17:00 on a sloppy channel, it might initially be "recording -0:05 +0:15 = 15:55-17:15 (duration 1:20)". Both the time and the offset field are editable, whichever method the user finds more conveniant.
• Text-based rules with priority for automatic scheduling .
  For instance: match dutch-spoken channels with the word 'teletubbies' or 'tele-tubbies' in the title before 18:00, and on a higher priority match anything with the word 'inspector' and 'morse' in title or description.
• Very-few-key-press(tm) simple direct recordings. Suppose you are watching a live (possibly shifted) show, but you suddenly want to record it: select the osd-menu (1 key), press record (1 key). The frontend shows the data-entry screen with the programdata (where you could correct the endtime if you wanted), pressing confirm (1 key) will suffice.
  Ready at the count of three keys.
• Simple partial manual recording from the time-shift buffer. Useful for example to save a single item from the news. Wind to the point were you want to start, press the record button. Wind to the point were you want the recording to stop, press the record button again.
  The recording will be saved with a recognizable description, which you can later change.
• Commercials or leading/trailing parts can be marked for skipping. Upon playing the player will jump over them, and on export they will be skipped to save diskspace.