Lessons Learnt 2 - Check the Quality

This is Part 2 of the Lessons Learnt series and follows on from the previous post.

I like how different fields of practice converge. For instance how patterns originated in architecture and were taken onboard by software and interface design practitioners.

Test Strategy

Recently I've been back involved again in writing a testing strategy for one of my clients. I have some quite rigourous views where there's a customer involved (there *will* be separate rounds of system testing and user acceptance testing with different testers and different test scripts), but flexible in terms of how long the testing phase will be and who has to perform it. A lot of it comes down to being pragmatic and managing risk. A lot of the time, the risk management is more about identifying what to do in case the risk does transpire rather than trying to prevent the issue in the first place. What if a major part of the system does not work as intended? The prospective user could continue as they are and not use the new application, they can continue while having the issues resolved or they can use the new application complete with the issues (potentially while having them resolved or not). Anybody who's been involved in this phase will know there's a lot more to it than that, but for the moment, I've said enough to start converging the two fields

Testing in the Field

For Memory, we did some test mic recordings. These weren't in the same conditions as the main recordings. Think of the mic tests being a minor proof of concept. As we progressed towards filming (but still before), we should have performed a more accurate proof of concept based on the filming conditions. We encountered two major issues (1) wind noise and (2) electrical interference from the DVcam's power supply.

Proof of Concept

As for (2), a proof of concept of recording speech in an enclosed space complete with some electrical equipment such as network, fax, phones and fluorescent strip lights would have helped us notice some of the issues. However beyond that we should have listened to the sound quality at the time of recording. I didn't listen to it since I wasn't in charge of location recording. Nobody was (well I guess Ian was by default since it was his project and he was director - however it'd be unfair to blame anyone for this - it was specifically created as a learning project for many of the members after all). It was one of those elements that fell through the gap. Any project, no matter how well planned, will have these gaps. Decent, experienced team members will either expand their role to cover the gaps often in advance of the gaps appearing or help organise the team (even it's only a question of referring it up the chain) to help fill the gap.

Mitigation

Testing mitigates some of the risk and reduces the effect of those gaps being present by the time the product is released. In our case, we were under pressure to complete filming due to limited availability of actors, locations and director. To give you an example, one of the actors was available for 3 hours maximum on one day, one of the locations only available for a few hours, etc, etc. These are all constraints on the filming process. For each filming day, we only just had enough time to get the shooting in, let alone do anything else. In the end, testing was minimal to the point of just listening to see that there was some sound and nothing major in the background. This did result in some scenes being re-shot, note the credit of "have you checked the sound?" - this was a major problem on the first day of shooting. However, we didn't listen with as much attention as the production would require. This project wasn't meant to deliver anything HD quality, but listen to the episodes and you can tell where we'd learnt lessons from previous recordings. There is a noticeable difference in audio quality between the sessions with the DV cam plugged into the mains to charge and those where we used it solely on battery (battery is much cleaner).

So we didn't do enough testing and it showed.

I managed to clean up the original recordings using a variety of filters, noise-gates, parallel compression, eq and automation of each channel. But it's still not clean.

The Results

End result, we:

have noisy background in the original recordings

recorded some of the voices again so the actors had to be available a 2nd time

spent a lot of time cleaning up the noisy background audio

spent less time writing music which is what I'd set out to do

How to Resolve

For software or process implementation, testing is a key part of the implementation. It should also be key for film-making, even at the lowest budgets, including zero-budget. In our case, I'd recommend an entirely different audio recording set-up, but failing that. For the first take in every scene, the audio should be listened to on headphones (preferably closed-back to reduce noise from the environment) by a single person responsible for saying whether it passes or not.

In parallel, someone should be responsible for the video content. Doesn't have to be two people, just understand that there are two roles. Then keep going until you get the takes that you're happy with. At that point, check those as well. This unit testing will give you more confidence that the product is to the quality you expect it to be. The trick is to maintain momentum and keep a pattern of film first take, check, progress through the takes. If everyone knows the order, then it will be easier to follow.

On a tangent from another discipline, facilitation, I'm a great believer in any method of getting a common order of events that can be referred to by anyone at any point. Flip-charts for this purpose seem to have caught on a lot in the places I work, but they're just one method

Priorities

The alternative to testing is mix of potential disappointment for everyone involved, inferior product or failure. When you think of it that way, then testing becomes easier to swallow.

Although I've discussed audio, a lot of the content applies to video. Was the white balance correct, what about backlight, contrast.

Wrap-up

So what's the lesson? Just like any project, ensure that you account for testing in the film shoot itself. And don't skimp on the time you allocate to it.