Author Archives: STEEPLE

AV workflow in the University of Reading

by Dr David Wong
The workflow provides for an end-to-end solution where users submit their audio / video file on a university web page, and after a few minutes receive an email containining instruction on how to link the transcoded media file on their web page.
The systems components consist of:

Workflow server (Ubuntu), provdes a platform for NSSDropbox 2 (from University of Delaware).
A modified version of NSSDropbox 2 using LDAP security, provides upload facility (2.0GB limit per file). Each submission causes an email sent to a designated email account
Script (Perl) in workflow server scans email account for new message,  extracts uploaded file out of NSSDropbox’s internal storage, writes it to a designated directory, emails administrator a notification of the upload, and emails the user instruction on how to link the transcoded files from their web page
Mac OS XServe, running Episode Engine, scans the designated directory regularly for new files, does transcoding, and writes transcoded files to an output directory
The output directory is mounted on a web server, making the files available on the web
Templates in University CMS, and VLE (Blackboard), enable end users to link the transcoded files from their web page
JW Player

File formats:

Audio: accept mp3, wav and wma, and output to mp3
Video: accept mpg, mp4, avi, and output to mp4 h264 AVC (Levels 1.2 and 3), and AAC audio encoding. Some success with other input formats wmv, tod, mod etc.

Advantages

Episode Engine’s preset formats and workflows transcode files that conform to h264 standards.
JW Player provides for a large number of platforms.
Very low load on servers. 
Apart from staff costs, main expenditures are XServe and Episode Engine.
Multiple workflows based on media type or requirements (e.g. teaching and learning, audio only, video).

Issues

Episode Engine is not easy to use, nor flexible; its warning and error messages can be confusing and far from instructional. In cases where it is stuck with a job that’s difficult to transcode, it could hang and requires intervention by aborting the job. It’s possible to modify / create workflows which requires root access, and I didn’t pursue that.
Episode Engine strips out metadata contained in file.
Workflow server (running NSSDropbox 2) is inside the firewall to minimise security issues. Off-campus users can upload if they log in using VPN.
Not really an issue, but for people new to Perl and scripting (or those thinking about implementing NSSDropbox 2), we did spend a lot of time (weeks) modifying NSSDropbox 2 code, adding extra bits (and removing/disabling others), and hours of testing. And time spent learning Perl.

Migration test case
Two years ago, most of the univeristy video files were hosted externally. With Episode Engine and Helix, we began hosting the assets in-house. By mid September 2011, we have about 700 AV files stored on a disk attached to the Helix streaming server (a majority of these originally migrated from another project dedicated to teaching and learning). Most of these files are in mp4 (video) or mp3 (audio) formats as for about 1.5 year we stop transcoding file in Real format and also ceased streaming, giving preference to wider standards. This paves the way for a relatively easy introduction of the “AV Dropbox” scheme (easy in the sense that we didn’t have to re-transcode the files)
Migration consists of requirements analysis with stakeholders, and commissioning of servers (workflow & upload using NSSDropbox, and web) and storage areas (Netapp, one for upload and one for transcoded files). Other issues came into play including the use of a “friendly URL” (we decided on avcontent.reading.ac.uk), and proxying the workflow (or not, being the motion at the moment), and re-transcoding a small number of old files to mp4 / mp3.
Going live was an event that demanded some added attention as updating the CMS and Blackboard templates (to point to new server URLs) meant currently available AV files needed to continue to work with the new URL. A great deal of checking went in to ensure going forward does not result in going back swiftly! Apart from some templates that were non-standard, the go-live went unnoticed and was a quiet success.
Next step
JW Player remains an attractive asset. With HTML5 (and mobile platforms) becoming mainstream, we are looking at what’s possible. There is significant requirements from the corporate office to provide for these outlets. A bought-in solution is unlikely, nor something that potentially gives security concerns. Cloud solution maybe possible, so long uploaded files are not stored on remote servers.
Zend.to appears to be attractive with regard to workflow.

Dr David Wong Multimedia: Corporate Information Systems Support IT Services, University of Reading

Permalink

| Leave a comment  »

Your technical help is requested…

Hello fellow Steeplers.
You may remember that I’m working on a project to improve reporting on the impact of podcasting (Listening for Impact @ JISC). I am in the process of writing an application to do a lot of the heavy data analysis from various sources (Apple spreadsheets, Apache Log files, Google Analytics, etc), but I’ve got a little bit of a challenge that I expect some of you can help with.
I am trying to identify a range of User Agents that appear in our hosting logs, but that aren’t identifiable using the tools I presently have. I have documented the latest on this on the LfI website and would ask you take a look at User Agent Analysis – Part 2: Name those agents to see if you can help us (and ideally save me a lot of time on Google and experimentation!).
Thanks!Carl

Permalink

| Leave a comment  »

What is Google playing at?

I said earlier it hasn’t been quiet lately. Another hot topic received a shove recently when Google announced they were removing support for the h264 video codec from their Chrome webrowser in favour of their own WebM format (oh, and Theora). This simply means that HTML 5 tag content will not playback without needing some third party software/plugin (where said content is in the globally dominant and arguably superior h264 format).
I can only conclude this is not a good thing.
Why? Well here’s a few observations resulting from a community discussion of this topic…

Desktop browsers that are not Internet Explorer or Safari still account for

Thoughts on Apple in Podcasting

Whilst it’s been a while since I last wrote, things haven’t been quiet. When Apple announced in Nov 2010 that Xserve was to be discontinued three months later, shockwaves went out through Apple’s enterprise community. Having gone through the tech equivalent of the 5 stages of grief and spent some time discussing this with colleagues near and far, I thought I’d jot down some notes on current thinking amongst us.

Apple have killed off the Xserve, it’s not coming back however we wail.
They have no current product that will suffice in a datacentre for any serious OS X based computing power/capacity (for various issues of density, resilience, connectivity, politics, etc).
There are only three services that exist only on OS X Server: Software Update, Netboot and Podcast Producer. Every other service has an equivalent or alternative on either Linux or Windows, and they have many acceptable options for datacentre deployment.
From a podcasting perspective, of those three, we’re only interested in Podcast Producer, so I can ignore the other two and rely on our talented SysAdmins to provide equivalent services for other functions (Directory, Apache, etc).
Podcast Producer has a range of faults/issues, including:

Capture is too unreliable and expensive to be deployed further than a pilot setup
The Library requires too much development work to be done on it to allow for edits to be made to metadata, etc
The workflow is QT dependant, and has a major bottleneck on the render task as that is reliant on GPU capacity (something Xserves are weak on)
To get a usable range of output codecs, you have to install alternative compression software/applications, several of which have workflow-process like capabilities duplicating functionality in PcP
There’s no indication these are going to be addressed/improved (see below on Apple and roadmaps)

Because of the above, Oxford has failed to seriously use PcP as intended from the outset, and we know we’re not alone.
To support a transcoding service scaled for university wide use, we created a cluster of Xserves backed by Xsan.
Xsan has been ok (1 fault in 2 years of operation – almost zero dataloss), but it requires qualified SysAdmin maintenance (not cheap or easy) and two Xserves (or at least, two machines with FibreChannel cards in, thus ruling out MacMinis in the datacentre) to act as Metadata Controllers. Whilst these machines can perform other tasks, it’s not recommended to put anything too intensive on them lest their response times decline.
Xsan has some weaknesses:

Only 85% of a volume capacity can be used before serious performance degradation kicks in.
This means that for a 16 x 1Tb drive chassis, two drives are given to Metadata, one is left as a common hot spare, and the remaining thirteen are in a RAID 6 configuration for speed and resilience. That gives us a Volume of 11Tb theoretical space, which after formatting is 10Tb. 85% of that is 8.5Tb of usable diskspace out of 16Tb of theoretical capacity, or if you prefer, little over half of the total storage purchased is usable!
Licences, whilst far cheaper than the Linux or Windows equivalent (Xsan is just Quantum’s StorNext FX SAN product rebadged), are not cheap, even on educational pricing, and you need one per machine attached.
FibreChannel is not a cheap technology to introduce (minimum £3000 investment), even more so if you want to attach video editing workstations to a central processing cluster.
All told, this make Xsan expensive to buy and expensive to maintain.

With Xserves now on a dying path, that gives Xsan about 2-3 years to live.
OS X Server is a separate issue from Xserve, but not entirely if your policy is that servers should operate in a datacentre.
There is no roadmap for OS X Server:

This isn’t news, Apple don’t do roadmaps in general, you take that as given whenever you buy from them, but this lack of future planning upsets enterprises whose strategies are often like supertankers and need long periods to turn around in.
There has been no announcement or even hint of OS X server existing beyond 10.6.n; again not surprising, but when you pull out the rug from customers with the end of an entire product line, it would help to reassure them their services might continue in another form. This is unusually bad PR for Apple, and lends credence to the idea Apple is withdrawing from the enterprise and/or education market to a degree.
If OS X Serve does continue, the only solution we can imagine that would make that make it viable in a datacentre is to virtualise it on non-Apple platforms. Indeed, this has been nicely campaigned for by Dave Schroeder (http://appleopenletter.org/). However, as Universities have been investing in virtualised infrastructure already, existing Mac hardware isn’t going to fit into that and thus becomes redundant anyhow, and Xsan becomes surplus to requirements also.

The above issues aside, the biggest problem from this announcement and subsequent (near) silence, is simply the trust that has been broken between Apple and clients that have bought into Apple’s enterprise products. This has been a vivid reminder of common Apple behaviour (consumers have known this for years) and undoes nearly all the work Apple have done in the 9 years of developing a server product/solution.
So, we conclude that:

Our Xserve investment is now EOL’d and we have to switch platforms. This is exceptionally unfortunate in current economic times where higher education is especially hurting.
Podcast Producer is no longer an option for centralised podcasting services and needs to be discounted from future thinking.
Xsan is redundant along with the Xserves (even though the hardware and tech will likely continue to run happily for the next 2-4 years – ignoring the security concerns of unpatched software).

As we’re now onto Acceptance, we look to the future… a future that may contain Matterhorn, Mediamosa, Cisco, Linux and more, but not Apple (apart from on a desktop or two ;-)
More on our future planning when we’ve finished evaluating our practical options.
Carl

Permalink

| Leave a comment  »

Listening for Impact

Little bit of cross publicity here…
JISC are currently funding the Learning Technologies Group at Oxford to do a quick investigation on the impact of podcasting. This is where my day to day efforts are currently focussed, and since I’m dealing with the technical approaches side to this analysis, some of you may be interested to learn what we’re doing. 
If Log Analysis, Analytics, SEO and Stats are your bag, pop across to the project website (Listening for Impact) and keep an eye on the postings over the next 5 months. We would appreciate your feedback.
Carl

Permalink

| Leave a comment  »

Workshop at UCL – 22-11-2010

For those tracking what’s going on, but not noticing the wiki or the mailing list, here’s a brief reminder of our (last?) Benefit’s Realisation Workshop for Steeple. We’re gathering at UCL on Monday 22nd November for a 1 day workshop to cover a range of topical issues. More details can be found at http://www.steeple.org.uk/wiki/BR/Workshop_Nov_2010_UCL.
See some of you in London :-)
Carl

Permalink

| Leave a comment  »

When Law meets Technology

http://www.theregister.co.uk/2010/09/30/downloads_performance_ruling/
The above story caught my eye in relation to a US court decision to term “downloading” as not equal to “performance”, but to say that “streaming” is a “performance”. It’s got me curious about the UK position on the matter now, but since IANAL, I’m not going to chase that one for a while.
I have the feeling that this judgement could make an interesting impact on damage calculations for say, a libellous (or is it slanderous?) recording. After all, if they’re podcasts and being downloaded, they’re not being performed, so has a person been damaged by them? But if it’s youtube, or a streamed output… well…
And if that logic holds, does this suggest that any perceived “risks” to an institution are less from podcasting than streaming?
Anyhow, the Steeple blog isn’t dead…it’s just a little quiet at the moment with my being busy on various other projects. Some more coming this way in the next few months though as I’m tackling stats and monitoring/management setups for institutional podcasting.
Carl

Permalink

| Leave a comment  »

New resources on the website

Greetings all!
We’ve recently (i.e. in the last hour) added some new resources to the Steeple website that could well be of interest to you:

A kit to help you launch your institution into iTunes U
Updates to the audio and video podcasts from the Open University
11 new audio interviews with various Steeple community members talking about their experiences in setting up podcasting at their institution 

Launching iTunes U Project Kit
This collection of slides, documents and notes was compiled by Apple in conjunction with Steeple and it’s members to provide a starters guide for anyone else looking to join the iTunes U public system. You can find the kit referenced in the Outcomes and Outputs section of the Project Report on the Steeple Website.
Quicklinks to download for:

MS Office
Apple iWork

Open University Podcasts
There are three video and three audio podcasts from the OU about their podcasting experiences, including a new audio podcast discussing issues of Rights management. You can find these files on the Steeple website in Resources -> Audio and Video -> OU Podcasts.
New interviews with community members
Another output from the Open University as part of the Steeple project is this collection of 11 audio interviews with various academic, support and administrative staff from 5 institutions within the Steeple community, where they discuss their experiences with podcasting and service creation within their institution.

Podcasting at Oxford: Overview
Podcasting at Oxford: Low Resource Podcasting
Podcasting at Cambridge: Range of products
Podcasting at Cambridge: New Steeple Portal
Podcasting at Coventry
Podcasting at UCL
UCL on iTunes U
Cambridge on iTunes U
Coventry on iTunes U
Oxford on iTunes U
Accessibility of podcasting

You can find all these under the Interviews heading on the OU Podcasts page within the Steeple Website (Resources -> Audio and Video -> OU Podcasts).
Thus concludeth my marketing message for you today. More news soon on the recent Workshops run through our Steeple Outreach programme (Steeple BR) including screencasts and notes.
Carl

Permalink

| Leave a comment  »

Keeping on top of progress

There’s much debate going on in multiple locations about the future of various video codecs and playback mechanisms. If you’ve not already seen them, I’d recommend a read through these two postings from Jeroen Wijering, creator of perhaps the most popular flash based video playback tool in use on the web.

http://www.longtailvideo.com/support/blog/11887/html5-video-not-quite-there-yet — talks about some of the other shortcomings in the html5 video tag specification, beyond the high profile badgering of the web browser vendors with regard to the codecs supported. The comments have a smattering of sensible inputs (and unfortunately a number of corporation hating FUD) but it does raise the issues over how HTML5 still needs to develop. 
http://www.longtailvideo.com/support/blog/12120/the-google-vp8webm-announceme… — this is a posting I read a few weeks ago and thought I’d passed on to you all already, but seems not. Jeroen is talking here about the launch of WebM and VP8 from Google, and how it compares with the current options. I’m not convinced by his excitement as to WebM becoming the de facto format in the near/medium term, but I think it is going to have a part to play. One thing is still clear, I’m not beating back the requests to implement our content in either Ogg or WebM/VP8. 

I mentioned the debates, well ignoring the official ones around the standards group, I’m paying close attention to those going on within the Terana TF-Media and the Opencast community (notably a discussion group being pulled together by Brian O’Hagan). I’ll report more on my observations and our actions as we move on.
Update – 1/7/10@11:32GMT
Seems I wasn’t alone thinking this and just catching up. YouTube have posted a near identical set of reasons as to why they’re not fully moving to HTML5 (yet?) and that appeared about 3 hours after the first posting of this (childish glee at timing? moi?)
You can see the blog at http://apiblog.youtube.com/2010/06/flash-and-html5-tag.html or you can read The Register’s summary and background at http://www.theregister.co.uk/2010/06/30/why_youtube_still_uses_flash/ .
Carl

Permalink

| Leave a comment  »

Podcast and Podcast Capture not signing in

Techie note that might be of help to some, and one I’ve just encountered.
Problem: Unable to publish a workflow to Podcast Producer 2 cluster – Client returned Unknown Error with a little yellow triangle warning.
Also found:

that I could not connect to the PcP2 server via Podcast Capture on either the host server or any remote machines; 
that the podcast cli was failing with:

LTGS-Macbook:zinstance ltgadmin$ podcast –installworkflow -s cynewulf.oucs.ox.ac.uk -u ADMINUSER -p PASSWORDHERE –path /Users/ltgadmin/Documents/Basic\ Single\ Test\ 1.pwf Packing workflow bundle: /Users/ltgadmin/Documents/Basic Single Test 1.pwf… Packed workflow bundle archive: /var/folders/T5/T5mWe7F+GXWdPTvQsSRQxE+++TI/-Tmp-/podcast-workflow-download-CD308355-34A8-4B0D-ACDB-F11369786B68-20100611-7686-10decbx/CD308355-34A8-4B0D-ACDB-F11369786B68.pwf.cpgz Installing /Users/ltgadmin/Documents/Basic Single Test 1.pwf… Destination server: https://cynewulf.oucs.ox.ac.uk:8170/podcastproducer Destination workflow: Basic Single Test 1 NoMethodError /usr/lib/podcastproducer/auth/auth_challenge.rb:37:in `auth_properties_to_hash’/usr/lib/podcastproducer/auth/auth_challenge.rb:31:in `each’/usr/lib/podcastproducer/auth/auth_challenge.rb:31:in `auth_properties_to_hash’/usr/lib/podcastproducer/auth/digest_auth.rb:22:in `generate_authorization_header_value’/usr/lib/podcastproducer/client/remote_command.rb:141:in `add_digest_authorization’/usr/lib/podcastproducer/client/remote_command.rb:393:in `execute_once’/usr/lib/podcastproducer/client/remote_command.rb:284:in `execute’/usr/lib/podcastproducer/client/remote_command.rb:281:in `each’/usr/lib/podcastproducer/client/remote_command.rb:281:in `execute’/usr/lib/podcastproducer/client/remote_command.rb:66:in `post’/usr/lib/podcastproducer/client/workflow_administration.rb:121:in `install_workflow’/usr/bin/podcast:1002 Remote command ‘https://cynewulf.oucs.ox.ac.uk:8170/podcastproducer/workflows/create_upload’ failed: undefined method `-’ for nil:NilClass

or something simpler:

LTGS-Macbook:zinstance ltgadmin$ podcast –listworkflows -s cynewulf.oucs.ox.ac.uk -u ADMINUSER -p PASSWORDHERE NoMethodError /usr/lib/podcastproducer/auth/auth_challenge.rb:37:in `auth_properties_to_hash’/usr/lib/podcastproducer/auth/auth_challenge.rb:31:in `each’/usr/lib/podcastproducer/auth/auth_challenge.rb:31:in `auth_properties_to_hash’/usr/lib/podcastproducer/auth/digest_auth.rb:22:in `generate_authorization_header_value’/usr/lib/podcastproducer/client/remote_command.rb:141:in `add_digest_authorization’/usr/lib/podcastproducer/client/remote_command.rb:393:in `execute_once’/usr/lib/podcastproducer/client/remote_command.rb:284:in `execute’/usr/lib/podcastproducer/client/remote_command.rb:281:in `each’/usr/lib/podcastproducer/client/remote_command.rb:281:in `execute’/usr/lib/podcastproducer/client/remote_command.rb:49:in `get’/usr/lib/podcastproducer/client/workflow_administration.rb:20:in `list_workflows’/usr/bin/podcast:861 Remote command ‘https://cynewulf.oucs.ox.ac.uk:8170/podcastproducer/workflows?language=en’ failed: undefined method `-’ for nil:NilClass

… but that the PcP2 web interface worked just fine with any valid username and password I tried.
Looking at the PcP2 logs I would see:
From attempts to login via Podcast Capture (on host server):

Processing WorkflowsController#index (for 163.1.13.145 at 2010-06-14 15:58:03) [GET] Parameters: {“action”=>”index”, “language”=>”en”, “version”=>”2″, “controller”=>”workflows”} Rendering default/failed.xml.builder (401) Filter chain halted as [:authorize] rendered_or_redirected. Completed in 32ms (View: 1, DB: 2) | 401 Unauthorized [https://cynewulf.oucs.ox.ac.uk/podcastproducer/workflows?version=2&langua...]

From successfully login via Podcast Producer Web Interface:

Processing CaptureController#login (for 129.67.101.46 at 2010-06-14 16:01:37) [GET] Parameters: {“action”=>”login”, “controller”=>”capture”} Rendering capture/login Completed in 10ms (View: 5, DB: 3) | 200 OK [https://cynewulf.oucs.ox.ac.uk/podcastproducer/capture/login]

Google and Apple Mailing lists have been my friend in terms of finding a working solution, however I’m not yet confident about the cause of the upset in the first place. We have been doing some work to create a working OD for our institution based off of our lack of a central directory service, but this should not have impacted on the Podcast Producer setup, nor the machines used to connect to the PcP2 service, nor the accounts used (as they’re locally created in the OD, not imported from outside). The Solution was to remove the Digest option from the http_auth_type keyset in the cluster_preferences.plist as described here: http://lists.apple.com/archives/Podcast-producer/2009/Oct/msg00007.html If anyone else has this problem, and can isolate what it is they’ve changed to their setup to cause this to stop working (as it all once did just fine), please let me know. Thanks, Carl

Permalink

| Leave a comment  »

Copyright © 2007 Institutional Innovation (p). All rights reserved.
Institutional Innovation