Partitioning Rollout

 Available in: Krakatoa v1.0.0 and higher.

1 Partitioning Rollout

Particle partitioning is a way to multiply the total particle count of a particle system through the repeated running of the particle system with different seed values. Generally speaking, particle partitions are related subsets of particles processed independently and cached to disk files.

Each partition is generated using different random seeds to generate a slightly different solution.

The partitions are combined at render time to produce an aggregate result containing more particles than 3ds Max can normally generate in a single pass.

In addition, partition generation can be optionally split between multiple machines using the Deadline Network Manager or by manually running multiple copies of Krakatoa on multiple workstations to generate the same high amount of particles in a fraction of the time needed to process in a single pass. For example, instead of creating 10 million particles per frame on one machine, you could save 1 million particles on 10 machines in 1/10th of the time.

Krakatoa contains tools to automate the process of creating particle partitions.

These tools are only available when Particle Render Mode in the Main Controls rollout is set to "Save Particles To File Sequence".

Partition Count

Partition Count controls the total number of partitions to be created.

Default value is 10.
Value Range is 2 to 100.
The number will be included in the saved particle file's name, for example three partitions of "particles_.prt" will be saved as particles__part1of3_.prt, particles__part2of3_.prt and particles__part3of3_.prt
For each partition, the Random Seeds of Particle Flow operators and Legacy Particle Systems will be incremented as discussed below.
NOTE: The first partition will always use the original Random Seeds, the second partition the original Random Seeds Plus One and so on. Thus, the content of Partition 1 would always be identical to the content of a particle file sequence saved without partitioning.

See Krakatoa FAQ for discussion of partition counts.

Partition Range From M To N

The Partition Range From and To spinners can be used to generate one or more partitions of the full range of partitions.

This can be useful when the partitioning had to be canceled prematurely after N partitions have been generated and you want to continue from a higher partition than 1, or to replace a single partition with a fresh file sequence.
To generate only one partition out of the whole range, simply enter two identical values in the From and To fields.

Random Seed Toggles

Particle Flow uses pseudo-random numbers to create variation in particle behavior. To change the results, you can enter different seed values to alter random number computation. The Random Seed field appears in several Particle Flow operators under Uniqueness: Seed, as well in the Legacy Particle Systems and some Thinking Particles operators.

When creating particle partitions, you can toggle the following options to select which seeds are automatically incremented:
- PFlow Position - affects Position operators like Icon Position and Object Position
- PFlow Spawn - affects Spawn operators like Spawn and Collision Spawn.
- PFlow Speed - affects Speed operators like Speed and Speed By Surface
- PFlow Rotation - affects Rotation operators like Rotation
- PFlow FumeFX - affects FumeFX operators like FumeFX Birth
- PFlow Others - affects any other PFlow operators that have a Random Seed value but are not in any of the above categories.
- Legacy Particles - affects 3ds Max legacy particles of second generation (Blizzard, PArray, PCloud, SuperSpray) which provide a random seed value. First generation Legacy systems (Spray and Snow) do not provide a random seed and cannot be partitioned correctly.
- Thinking Particles - affects Cebas Thinking Particles operators that have a random seed.
- Box#3 Geometry - affects Geometry sub-operators inside Orbaz Particle Flow Tools Box #3 Data Operators and Tests.
- Box#3 Random - affects Random sub-operators inside Orbaz Particle Flow Tools Box #3 Data Operators and Tests.
- Box#3 Sclr/Vector - affects Scalar and Vector sub-operators inside Orbaz Particle Flow Tools Box #3 Data Operators and Tests.
- Box#3 Icon - affects Icon sub-operators inside Orbaz Particle Flow Tools Box #3 Data Icon Operators and Tests.

Note that in addition to the built-in Seed Incrementation, Krakatoa provides a way to execute arbitrary MAXScript functions defined by the user as described here.

Excluding Seeds Of Specific Objects, Events and Operators

Krakatoa allows you to control the seed incrementation at a finer level using a special naming convention for objects, Particle Flow Events and Operators, as well as Thinking Particles Systems and Dynamic Sets.

If an Object Name contains the string "NOSEED" (case-insensitive, "noSeed", "NoSeed", "noseed" etc. would also work), the object will be skipped even if it matches a category that is enabled for Seed Incrementation. The objects include Particle Flow Emitters and Legacy Particle Systems with Random Seeds, as well as Thinking Particle systems.
If a Particle Flow Event contains the string "NOSEED" in its name, all operators inside the event will be excluded from Random Seed incrementation.
If a Particle Flow Operator contains the string "NOSEED" in its name, the single operator will be excluded from Random Seed incrementation.
If a Sub-Operator inside a Orbaz Particle Flow Tools Box #3 Data Operator or Test contains the string "NOSEED" in its name, the single sub-operator will be excluded from Random Seed incrementation.
If a Thinking Particles Dynamic Set contains the string "NOSEED" in its Comments field (the edit text field on top of the set), any Operators inside the Dynamic Set will be excluded from Random Seed incrementation.

In addition, if the >Affect Seeds of Selected Objects Only option is checked, any objects that are not selected in the scene will be excluded from Random Seed incrementation. This is supported in both local and network partitioning mode - the current selection state will be recorded and submitted with the Deadline job.

Combining Partitioning with Particle Flow Toolbox #3 Disk Cache Operators

The Disk Cache operator included in Box #3 and also sold as a stand-alone tool can be used as a "safety net" when performing Partitioning. It can cache particles to disk files while the Particle Flow is updating and reuse the saved data if the Flow has to be recalculated again from the beginning, for example after a crash or power loss in the render farm.
When the >Increment Box #3 Disk Cache Filenames option is checked, Krakatoa will change the output file names of all Disk Cache operators to match the same partitioning naming convention as Krakatoa partitions. If the Box #3 is set to cache "Render", cache files with the original file name plus the partition signature will be saved to disk. If the partitioning has to be canceled and continued at a later point, all frames that have already been partitioned will use the cached data instead of calculating everything from the first frame, resulting in much shorter pre-roll times.
Once a partition is finished, the original file name will be restored.
Like with Seeds, adding "NOSEED" to a Disk Cache operator's name, event or flow will exclude it from being affected by Krakatoa partitioning.

Local Partitioning

Krakatoa allows you to calculate partitions on your local workstation. Of course, other than calculating partitions on Deadline, this option requires the partitions to be saved sequentially, which means a longer processing time.

The >Skip Existing Files option only affects local partitioning. When checked, Krakatoa will not save frames which already exist on disk. For example, if you were saving a partition and canceled in the middle of the processing, instead of setting the output range to start at the first missing frame, you can just check this option and save the original time range. Any existing frames will be skipped and only the missing ones will be saved. Since PFlow has to preroll to the first missing frame anyway, there is almost no difference in processing time, but it is much easier for the user. Also, if you want to update only a couple of frames from an existing sequence, you can delete them from disk and then run a partition with >Skip Existing Files on and only the deleted frames will be saved again.
The Generate Partition Range Locally... button will save only the Partitions defined by the Partition Rnage From ... To ... values. The name of the partition files will still use the Partition Count as the highest count, but only a subset of all possible partitions will be processed. For example, if you are saving 10 partitions (Partition Count is set to 10) but you want to process only partitions 2 and 3, you can enter 2 in the Partition Range From field and 3 in the To field and press this button. The sequences will be named ..._part_2_of_10_... and ..._part_3_of_10_... and you can process the rest of the partitions (1 and 4 to 10) later using the respective From and To values.

 Tip: If your workstation has 2 or more CPUs or Cores and you want to partition in parallel 
 without using the Deadline network manager, you can launch multiple copies of 3ds Max, load
 the same scene and save multiple sub-ranges with the same Partition Count, for example
 one copy of 3ds Max could save the range from 1 to 5 and the other from 6 to 10. Since Particle Flow
 is single-threaded, each CPU will be fully loaded and the only limitation would be the memory.

The Generate All Partitions Locally... button will save all partitions from 1 to Partition Count, completely ignoring the Paritition Range settings. If you have set the Partition Count to 10, Krakatoa will process from 1 to 10, if it is set to 5, Krakatoa will process from 1 to 5 and so on. This option is useful when leaving your workstation unattended and you want all partitions calculated in sequence.

Partitioning One Frame Only

In almost all cases, the user wants to partition a file sequence.
Very often though, the Time Output settings in the Render Scene Dialog of 3ds Max are set to a single frame.
In addition, saving a single frame while partitioning is not really possible - a "feature" in Particle Flow prohibits the changing of Random Seeds when a single frame has been requested for saving. Krakatoa has to save at least TWO frames to work around this limitation, so saving frame 0 results always in saving both frames 0 and 1.
To make sure the correct time segment is partitioned, Krakatoa 1.1.0 introduces a Sanity Check for single frame partitioning:

Local Partitioning Progress

During a local partitioning, a set of two progress bars and two text lines will appear in the area between the buttons and the Deadline Settings.
The top bar will show the progress of the current partition in red.
The bottom bar will show the total progress of the partitions in blue.
The first line will show:
- The partition Number
- The partition Count so far out of the total requested partitions count
- The requested partition Range
- The current Frame number
- The frame count so far out of the frame count for the current partition
- The current total frame count out of the total frame count for all requested partitions
- The total number of frames left to save.
The second line will show:
- The Last Frame Time in Hours:Min:Seconds
- The Elapsed Time so far in Hours:Min:Seconds
- The Remaining (Estimated) Time based on the progress so far.
- The text "Press Esc To Cancel."
You can press the Esc key at any time during the local partitioning (as long as 3ds Max has focus) to stop the process.

Deadline®

Krakatoa takes special advantage of Frantic Films Deadline render management system.

NOTE that Deadline 2.5 and higher provides a FREE TWO NODES MODE without any other feature limitations. Krakatoa could be used with Deadline running in FREE MODE to facilitate partition generation.

While Krakatoa can submit to Deadline 2.5, it is recommended to be used with Deadline 2.7 or higher as it takes advantage of some new features like running multiple Concurrent Tasks on one multi-CPU machine.

Submit All Partitions to Deadline

Click this button to submit automatic partitions to Deadline. Krakatoa will create the number of partitions specified in the Partition Count field, and will automatically name files based upon the base name defined in the File Picker. Random number seeds will be incremented based upon which seed buttons are toggled on and off. Job priorities and render pools will be based upon settings in the Submit Max to Deadline dialog.

Submit Current Partition to Deadline

Click this button to submit the current particle system to Deadline as a partition.

The resulting particle file will be named according to the Current Partition #, Partition Count value and the base file name entered in the File Picker in the "Save Particles To File Sequence" text field in the Main Controls rollout.

Deadline Job Controls

Krakatoa uses the functions provided by the Submit Max To Deadline (SMTD) script shipping with Deadline. It also exposes all relevant controls to the Krakatoa: Particle Partitioning rollout:

Job Name: this field will be automatically filled with the current .MAX file name.
- If the name has changed due to a new scene being opened, you can press the [>>] button to get the current MAX file name or base the job name on the save file name.
- You can also enter any name you want in the field.
Comment: this field will be automatically filled with the Max version and Krakatoa version number and a note that this is a Partition Submission job. You can enter any text you like in this field though.
Name: this field will contain the name of the user currently logged in Deadline. You can enter any other user name here.
Department: this field will be taken from the current SMTD script settings (if available). Changing the Department in SMTD or Krakatoa will automatically change it in the other script. This is the ONLY value shared between the two scripted GUIs.
Category: this drop-down list can be used to specify an existing user-defined Category. Categories are created via the Deadline Monitor in Super-User Mode.
Pool: this drop-down list will contain all pools available on Deadline. You have to select a valid pool for the submission to succeed.
Render Only On Machines In Pool: when checked, only machines assigned to the Pool will pick up the Partition jobs. When unchecked, any idle machines on the network will attempt to save partitions.
Priority: This spinner and color slider can be used to specify the priority of the job - the higher the number, the higher the priority.
Render Task Chunk Size (Frames Per Task): this value is set to 5 by default and specifies how many frames will be created within a single task.
Limit Number Of Machines Rendering Concurrently: this value is set to 1 by default and specifies the number of machines to work on a single Partition job. Since particle calculation is history-dependent, it is advisable to leave only one machine to calculate the whole partition job.

Note that the Machine Limit will be enforced (other than in SMTD where it can be disabled completely) and the Sequential Rendering flag will be set (if supported, in Deadline 2.6 and higher) for faster particle calculation.

If Deadline has not been detected on the machine running Krakatoa, a note describing Deadline and a link to the FREE MODE download will be displayed in place of the above controls.
You can use the Generate Partitions Locally options to get the same results, but in much longer time.