Working in Maya with Shelves and Scripts in a More Efficient Way

Optimizing the Shelf Panel

Sooner or later every artist inevitably adds so many tools to the shelf panel that it becomes overloaded, and it moves the scroll bar to the rightmost side of the shelf. From this point on using the shelf becomes unhandy. Not only does the artist need to scroll the bar for ages, but they are also unable to see all their instruments at once. Of course, you can create another shelf and switch between the shelves each time you need to use some of your instruments, but what is the point in buying big high-resolution screens if we cannot see everything we want on them?

Though, even if you have got only a 1080p screen, two rows of the most frequently used tools are always better than one.

Maya does not allow you to choose how many rows you need to see at once. So if you go through the menus and try to choose the relevant option (as we usually do to customize Maya to our liking), you will not be able to adjust the number of visible rows. However, Maya is very strong in its ability to use two programming languages to change anything we want – MEL (stands for Maya Embedded Language) and Python.

Because everything I talk about in this article relates to MEL scripts I would like to shortly describe three ways of executing MEL commands and scripts in Maya (those also apply to Python scripts, the only difference is that you should enter Python commands in the Python input field rather than the MEL field).

The first possible way to execute the MEL script is to type or paste it into the Command Line panel underneath Viewport and press Enter.

The second way to execute the MEL script is to use Script Editor which you can launch by pressing the rightmost button on the Command Line panel.

This can also be done by using Windows –> General Editors –> Script Editor menu.

The Script Editor will pop up then. Any script could be written or pasted into the bottom part (Input) and then executed by pressing Enter or the play button at the top. 

The third way is saving the script as a custom button on the shelf panel. It is way more convenient than entering the script and launching it each time you need to use it. The number of buttons on the shelf panel available for immediate use is important. To save a custom script as a button on the shelf panel you need to type or paste your script into the Input field (bottom part) of Script Editor and then press the “Save Script to Shelf…” button on the top panel of the window.

To increase the height of the shelf panel you may use the following script:

layout -e -height 97 ShelfLayout;

Here is an explanation of the script to help you fine-tune it for your needs. This line defines the height of the shelf panel in pixels. The default value is 45 for one single row of buttons. The value 97 provides the ability to use two rows of buttons without having to scroll at the cost of taking additional space above Maya’s Viewport window. If you want to use an army of buttons and your screen size and resolution allow you to do that, it is possible to increase this size even further up to whatever value you might need.

However, next time you launch Maya, the default value for the shelves panel height will be set. To override the default value on each Maya startup it is necessary to create (or, if you have got one already, to add a script to it) a file named 'userSetup.mel', which is a user script file for Maya, and then put the script in it. This file should go into the C:\Users\USERNAME\Documents\maya\MAYAVERSION\prefs\scripts
directory, where USERNAME is your local account name and the MAYAVERSION is the Maya version you use.

If you create this file in Windows Notepad make sure that “Save as type” is set to "All files". Otherwise, Notepad will add a .txt extension to the file without even notifying you and the resulting file will be of no use. The file extension should be .mel at all times.

After saving the file you will need to restart Maya to apply the changes.

Maya Shelf Dividers

To make use of the shelf buttons easier Maya allows to group buttons by their functionality using separators. One can notice that by looking at standard Maya shelves.

However, we cannot add separators to the custom Maya shelf through the menu. This is unfortunate because most users who work with Maya professionally use their own set of tools located on their custom shelf. Though, some more magic with text files can get us the separators on the custom shelf as well.

Namely, we have to open our custom shelf file located in
C:\Users\USERNAME\Documents\maya\MAYAVERSION\prefs\shelves.
You may open it in Windows Notepad or any other text editor which saves basic TXT files without advanced formatting. The file we need corresponds to the name of your shelf in Maya. For example, I do have a “MyShelf” custom shelf in Maya so I need to find the 'shelf_MyShelf.mel' file in this directory. When you open the file you need to add the following text lines

    separator
        -enable 1
        -width 12
        -height 31
        -manage 1
        -visible 1
        -preventOverride 0
        -enableBackground 0
        -backgroundColor 0 0 0 
        -highlightColor 0.321569 0.521569 0.65098 
        -style "shelf" 
        -horizontal 0
    ;

to the end of the file but above the last symbol “}”.

The number of times you insert this text one after another equals the number of separators you will get at the end of your shelf. I always add some more extra separators because I might need them later on. After making these changes you will need to restart Maya if it is currently launched. The separators can be moved between the buttons using the Shelf Editor.

You have to select the separator and move it with top/bottom buttons.

If you know a little bit of scripting or programming you might notice that you can paste the lines defining the separator right between the required buttons in your shelf text file. The same approach works well for the UV Editor custom shelf 

Note, however, that you will no longer be able to use Shelf Editor to move separators between buttons. You may use another approach that is also applicable to the main window shelves in Maya. You can drag the icons on the shelf with your mouse so that the icons are distributed between the separators.

Another Way of Adding New Shelf Functions

Sometimes we need to use certain Maya functions located somewhere deep inside an entire maze of menus. Repeating the menu sequence many times during a short period of time could become a hassle. In order to avoid that in the majority of cases you might save a MEL command called by the menu item as a button on your custom shelf. It is not possible to list all the commands associated with menu items so this part of the article is not about giving you a ready-made product but rather showing how you may find commands you might need on your own and then save them as the buttons to your shelf.

For example, we need to create a button on the custom shelf which will turn on Multi-antialiasing and Smooth wireframe in the scene in one single click. The first step is to understand what commands are executed when a certain Maya menu item is activated. We need Script Editor for that matter. Script Editor consists of two main windows – History (top) and Input (bottom). When Script Editor is launched we need to enable “Echo All Commands” from the “History” menu in order to see all the commands in History (in some menus commands are not shown in History automatically without this option being enabled). The Script Editor window should stay open until we add a new button to the custom shelf.

The second step is to clear Input and History. It is always handy to work with tidy instruments because then no command will be executed unintentionally. For that, we should press the “Clear All” button.

And here comes the action. While activating the function we need to save on the shelf, we need to watch the lines appearing in History. All the events will be shown there. Along with the MEL command itself, other code lines may also appear so we will have to further investigate what we need to use from the History and what is unnecessary. If you have some experience in scripting or programming, basic logic will help you to understand which line of the code is the command we need. Also, after playing around finding appropriate commands and assigning them you will gain some skill in separating the commands themselves from additional information. In some cases, it is necessary to experiment by putting every line in the History to Input, executing it, and analyzing what the execution of this line gives us.

A legal reservation has to be made here: neither the author of the article nor the site owner this article is being published on is liable for any negative consequences or damage caused by your actions related to the execution of any lines of code on any computer whatsoever. By following this advice you acknowledge that you understand what you are doing and therefore are fully responsible for your actions.

After all is set, let us activate the Smooth Wireframe function and see the History.

In this particular case everything is simple. One click of a button to activate a function – one command to execute – save as a button on your custom shelf. So let us do that. Just select the whole line with the command in History and press the “Save Script to Shell…” button. It will prompt you to enter the button name, programming language (choose MEL as Maya shows MEL commands output for its own functions) and save it as the last button on your shelf. After that, you can use this button and edit any of its properties in the Shelf Editor.

If you find some useful script on the Internet you can paste it into the Input field (bottom part of the window) and then select the language the script is written on, and finally select the whole code you have pasted and save it as described above.

What useful information can we get out of this command? This command has boolean data type as a parameter (no cut, add or intersect this time) meaning it can be true or false, in other words, it is either on or off. You can see it by looking at the “1” or “0” at the end of the command. Number “1” generally means “yes” in Maya while “0” means “no”. We know that from using the Channel Box panel.

To make the wireframe look harsh by turning off smoothing just replace “1” (true) at the end with “0” (false) and execute command. You can replace 1 with 0 and save another button on your custom shelf now.

But what if you need to turn on a smooth frame and antialiasing simultaneously, with the same button? Then you just need to find an antialiasing command and execute it right before or after the 'smoothing the wireframe' command with a single button click. For that, you should temporarily copy the first command from History and paste it into the Input, then press the Shift+Enter combination to move the cursor to the second line without executing a command. By pressing the Enter key you will execute the command you have in Input so be careful.

Now we have to clean the History. This could be done by pressing the “Clear History” button on the top.

Next, we activate Multisampling Anti-aliasing and see the History.

One of the lines starts (setAttr – set attribute) and ends (1 – a boolean type data) similarly to the previous command. So we can copy this code line in the History window and paste it into Input right below the first command. Then we can select them both and save them to the panel as we did with only one line of code. If you just press the “Save Script to Shelf” button without selecting the text, the content of the Input field will be selected as a shelf button.

Actually, you can gather whatever number of commands you need and save them as one single button on your custom shelf. You can find out which command is assigned to any Maya menu item, just like in the panels menu (but you cannot put the items from the panels menu on the shelf with Ctrl-Shift-LMB combination). You can even find out what command is used to change settings in the Preferences window so you can easily switch, for example, from Y Up to Z Up, between meters, centimeters, and inches in one single click. You can also make presets for clipping planes and switch between them in one click.

Certainly, there are limitations, not all of the commands are shown in History upon activating Maya functions. For example, it is possible to find the command activating Multi-antialiasing but there is no way Maya will show you the command to change its sample count. And I can give you some more examples and luckily there are not too many.

Another limitation that is for the majority of functions you will only get the boolean type of commands, not toggling, meaning that you will have to switch between two states of a function with two buttons (true/false, enabled/disabled). It would be easier, for example, to use one single button to change the X-Ray state from on to off and vice versa than use two separate buttons for each on and off state.
However, there is a way out in both situations – you can find almost anything on the Internet, particularly on subject forums, Autodesk knowledge base, and YouTube. I will also share with you some of the commands and scripts I consider useful and which I have found on the Internet.

Hidden Commands

Here are some of the scripts and basic commands that are not being shown in the History of the Script Editor.

polySelectEdgesEveryN "edgeRing" 2

This command allows you to select each second edge loop (if the edge loop is selected) or each second edge ring (if a single edge is selected). You may use any other number instead of “2” and it will get the job done for you accordingly.

setAttr "hardwareRenderingGlobals.lineAAEnable" 1;
setAttr "hardwareRenderingGlobals.multiSampleEnable" 1;
setAttr "hardwareRenderingGlobals.aasc" 16;

This script enables Smooth wireframe, Multisampling antialiasing, and sets the count of antialiasing samples to 16. Maya 2020 supports a 16-sample count, but Maya 2018 only supports an 8-sample count based on its Viewport 2.0 settings.

currentUnit -linear "cm"
currentUnit -linear "m"
currentUnit -linear "in"

This one sets the measurement units in Maya to centimeters, meters and inches correspondingly. Useful to swiftly meet project requirements.

string $selectedPanel = `getPanel -wf`;
int $shadedWireState = `modelEditor -q -wos $selectedPanel`;if(`modelEditor -ex $selectedPanel`)
{
    setWireframeOnShadedOption (!$shadedWireState) $selectedPanel;
}

Toggles showing wireframe on shaded on/off.

string $visPanel[]=`getPanel -vis`;
int $mode;
if (size($visPanel)>0)
{
for ($panel in $visPanel)
{
if (`match "^modelPanel" $panel`=="modelPanel")
{
$mode=`modelEditor -q -xray $panel`;
if ($mode>0)
modelEditor -e -xray 0 $panel;
else
modelEditor -e -xray 1 $panel;
}
}
print ("\nxray="+$mode+".");
} else
print ("\nNo model panels to modify.");

Toggles X-Ray mode on/off.

Scripts and Commands Outside of the Maya Menu

polyDuplicateAndConnect

This command is not available through Maya menus. It creates a duplicate of an object similar to Instance with one single difference: you can change the original object and any modifications will be duplicated on the linked object but no modifications on the linked objects will be duplicated on the original object unlike simple instancing.

polyCut

This command is also not available through Maya menus. It creates virtual cutting plane for the object. It can also cut objects not being joined together physically in any way if they are combined. Virtual cutting plane is operated by the tool properties in Cannel Box.

global float $vertexNormal[];
string $vertex[]=`ls -sl -fl`;
$vertexNormal = `polyNormalPerVertex -q -xyz $vertex[0]`;

Copy vertex normal.

global float $vertexNormal[];
string $vertex[]=`ls -sl -fl`;
for ($i=0;$i<size($vertex);$i++)
{
polyNormalPerVertex -xyz $vertexNormal[0] $vertexNormal[1] $vertexNormal[2] $vertex[$i]; 
}

Paste vertex normal.

select -cl;

This is a useful one if you need to deselect any selected object or its components. The difference between this function and Alt-D is that if any components are selected this command only deselects the components selected without quitting to the object mode. Alt-D not only deselects any components selected but also switches to object mode instantly.

Useful Scripts and Commands From Maya Menus

I decided to include some of the commands that anyone can find with ease but why would you waste your precious time searching for them if they are listed right here. Hope this helps you in your work.

{ string $objs[] = `ls -sl -type transform -type geometryShape`;if (size($objs) > 0) { xform -cp; } manipPivot -rp -ro; };

This command resets gizmo position to its default state. Equals to pressing Reset button in Tool Settings of the Move, Rotate and Scale tools. Especially useful to have it ready when changing pivot point frequently.

setAttr "hardwareRenderingGlobals.ssaoEnable" 1;
setAttr "hardwareRenderingGlobals.ssaoAmount" 0.692308;
setAttr "hardwareRenderingGlobals.ssaoRadius" 7;
setAttr "hardwareRenderingGlobals.ssaoFilterRadius" 2;
setAttr "hardwareRenderingGlobals.ssaoSamples" 32;

Turns on slight Screen Space Ambient Occlusion.

setAttr "hardwareRenderingGlobals.ssaoEnable" 0;

Turns off Screen Space Ambient Occlusion.

setUpAxis "z";

Instantly sets the Z-axis Up. Do not forget to reset the camera with the Alt-Home key combination in order to move correctly within the scene. Useful to swiftly adjust your work to project requirements.

setUpAxis "y";

Instantly sets the Y-axis Up. Do not forget to reset the camera with the Alt-Home key combination in order to move correctly within the scene. Useful to swiftly adjust your work to project requirements.

setAttr "perspShape.focalLength" 35;

Instantly sets the Focal Length of the camera to 35 (default). 

setAttr "perspShape.nearClipPlane" 3;
setAttr "perspShape.farClipPlane" 40000;

Instantly changes the near and far clip planes. Can be used separately or together as a preset to swiftly adjust your work to project requirements.

ExportSelection;

Instantly opens the standard Export Selection window.

Binding the Scripts to Maya Hotkeys

In order to efficiently use certain scripts, it would be wise to assign a hotkey for them. For example, the X-Ray function is much more useful if there is a possibility of turning it on and off within a second or less. The same goes for the 'deselect components' function. The hotkeys for any basic command and/or complex script could be assigned in Hotkey Editor.

First, you need to select “Custom Scripts” from the “Edit Hotkeys For” dropdown menu. Then you need to activate “Runtime Command Editor” on the right side of the window. Press the “New” button, fill in the fields as shown in the example below and press the “Save Runtime Command” button in the right bottom corner. You will have your script displayed on the left side in the “Custom Scripts” field. Now you can easily assign a hotkey to the script, save and close the window.

P.S. Just in case something goes wrong it is easy to return Maya back to its default state (in terms of configuration files) by closing Maya and renaming the following directory by adding the “_corrupted” suffix.

So it should go from looking like this: C:\Users\USERNAME\Documents\maya to looking like this: C:\Users\USERNAME\Documents\maya_corrupted.

Later on, you can easily transfer some of your shelf files or any other configuration files to your new folder which will be created by Maya upon the next launch.

I hope my guide will help you to use Maya more efficiently.

Victor Andreyenkov, 3D Artist at Ulysses Graphics 

Interview conducted by Arti Sergeev