Ten Ways To Simplify Your Render Pipeline with Qube!

This article first appeared on 3D Artist on 20 December 2012, but we thought you would find it interesting if you are looking at your render pipeline and more specifically Qube!…

PipelineFX, creator of Qube!, has kindly offered 3D Artist readers ten handy tips to simplify their render pipeline. Read on to learn more.

Ten Ways To Simplify Your Render Pipeline with Qube!
Thanks go out to PipelineFX for these very handy tips. Learn about the render farm management system Qube! and their other products over at the PipelineFX website.

1. Stick to one (or two) operating systems

Many 3rd-party applications have issues running on multiple operating systems:

  • not all plugins are available on all OS’s
  • fonts may render differently across OS’s.
  • bugs may exist in the same app on one OS and not the other

File permissions can be tricky to resolve across different OSs.

User identities may not be consistent between OSs.

2. Solve cross-platform issues first.

Each operating system accesses files in its own, unique way. If an artist submits a job from one operating system, but the facility has chosen to render on a second OS, the paths to the source material may be different.  For example, in an OS X environment, an artist may use a file located at /Volumes/fileserver/project/myproject.aep.  If the workers are running Windows, they would see the same file as either \\fileserver\project\myproject.aep or Z:\project\myproject.aep.

These types of issues can be solved using Qube’s path translation mechanism or they can be solved at an operating system level using symbolic links. The latter solution provides greater transparency and is less prone to paths that are outside of Qube’s control – like those defined within the source material itself.

3. Centralize your storage.

Artists’ work should not be kept on local disks. In order to render on all machines in a farm, each machine needs to be able to access all the files. Artists used to working on local disk often will add a “publish” step where they will copy their work to a central location when it’s time to render, but this is very error-prone and assets, textures, and external references often get left behind during the publish step. However, if artists get in the habit of working on shared central file storage, they will eliminate the necessity to publish. The other benefit is that their work can be easily transferred to other artists, and the risk of loss due to hard disk failure is minimized; central file storage is also usually configured to be much more reliable and is backed up more often than workstations’ local disks.

4. Open up file system permissions.

If your environment allows for a more lenient security policy, it may be in your best interest to set up the access permissions on areas of your file server where the artists are working so that all users have read/write access, instead of having to manage permissions on a more granular basis. Many sites work off the assumption that if you’re logged in, you should have access to almost everything on the server. Instead, adopt an ‘opt-out’ model; grant full access by default, then remove permissions from areas of the file system that require additional security. This is much easier to maintain than an ‘opt-in’ model, where the default access permissions are “no one accesses anything,” and permissions are granted on a case-by-case basis.

5. Centralised staging of worker/artists workstation for guaranteed consistency

Every computer that renders needs to have the same functionality as those that create the files to be rendered. They must have all the same applications, plugins, licensing, and OS settings. To guarantee consistency, we recommend basing all machines off a single, central machine image.

6. Use revision-based workflow (saved revisions of working files)

Rendering is an iterative process. One iteration is chosen as the preferred and selected for final. Using revision based workflow allows an artist to quickly move from one iteration to another, using the exact same settings as the approved version.

7. Ensure all dependent media is located on shared storage.

Artists should be instructed to actively work off of shared storage, instead of trying to move assets, textures, etc to shared storage when it’s time to render. The slower response some perceive when working off networked storage is many times less than the time lost to trying to get things up to shared storage at render time, and it avoids lost hours of re-rendering due to missed textures, etc. If your central storage is slow enough that the increase in load times experienced by the artists in their day-to-day work is significant, that storage will be crushed by the load imposed on it by a render farm when all machines start on a job at the same time.

8. Single centralised licensing server

Many applications use a central license server & depend on the availability of that server to determine whether or not they can start and run. Using a single license server allows for better accounting of license usage and is an easy place to track down problems if/when they arise.

Coincidentally, a Qube! supervisor will happily coexist on the same hardware as your license servers (assuming the server is up to spec).

9. Use Time-based cleanup scripts

Visual effects can be grueling work. Hundreds of iterations may be used on a single shot.  Iterations that are many days and/or weeks old, or even older, are of little use; often ending up more as a burden than an aid. Time-based cleanup scripts can eliminate the waste. Qube! provides API calls to help determine which jobs were submitted or completed more than X days ago to make the creation of these types of scripts that much easier.

10. Use commercial off the shelf products every place you can.

Don’t write what you can buy. The flip side of this is: “Don’t buy what you can’t customize.” A 3rd-party product that allows you to customize it to your workflow can provide the best of all possible worlds.

  • It’s easy to underestimate the cost and time involved in developing your own in-house solution; chances are someone else has solved the issue in a commercial product.
  • On-going maintenance, bug fixes, feature requests, etc, are never-ending.  Chances are that in-house expert who’s writing your custom solution isn’t cheap, and you hired them to work on something else.
  • Knowledge sharing is pretty rare for custom solutions; what happens when (not if) the developer who wrote your in-house custom tool leaves?
  • Most custom solutions are written under time pressure, and may not be the easiest code to maintain or understand, especially if the maintainer is not the original author. (see the previous point about employees leaving).
  • When something is written in-house, the tendency is to continually add features to it (these are free, right?), when these features might not be strictly necessary from a business standpoint. Your company gets paid to deliver work to clients, and these “nice to have” features cost money and time to produce, but often don’t pay for themselves in increased productivity. Users will learn to adopt and adapt to off-the-shelf products, and can often produce the same amount of work for the same effort with them.

Ten Ways To Simplify Your Render Pipeline with Qube!
This entry was posted in News, PipelineFX, Qube! and tagged , , , . Bookmark the permalink.