At some point in any motion graphic project, you will need to use one or multiple After Effects mask on your composition and I have to say even after 10 years of motion graphic experience I am still learning new tips and tricks to work with masks. I’ll show on this blog the basic to the advance techniques to use After Effects mask, so by the end of this article you will know how to use and more be comfortable with the mask shape and mask layers in After Effects.

Let’s start with the basics

What is an After Effects mask?

Essentially a mask in After Effect is a shape that hide or show a specific area of a layer (or multiple layers) on your composition.
Specifically a mask is a path / outline made of vertices that you can adjust or/and animate.

Type of masks in After Effects?

There are 2 type of masks you can use in After Effects
– regular mask
– shape layer mask.

After Effects mask compare to shape layer mask

TIPS: Something you must know when you create a mask 
If your select your layer on your timeline: You will create standard mask
If your don’t select your layer on your timeline: You will create a shape layer mask

Regular mask are often used for rotoscoping and morphing because you can quickly mask multiple frame.

How to create a mask in After Effects?

– Select your image/video on your timeline
– Top tools bar, click on the shape tool icon after effects shape tool icon (Shortcut: Press Q)
– In you composition draw your rectangle mask on your image/video

After Effects regular Mask demo example below

TIPS:
If you want to turn ON and OFF the visibility of your mask outline and shape path, you can use the Toggle Mask and Shape Path visibility icon Toggle Mask and Shape Path Visibility icon
located on the bottom of your composition.

How to modify a mask in after effects ?

There are many ways to modify after effects masks I will go over the essential below

Move one mask point at a time

Select the layer on your timeline
Press V (Selection tool)
Start moving each point

Move Multiple mask points at once

– Select the layer on your timeline
– Press M (Shortcut to access the mask properties)
– Select Mask Path
– Press V (Selection Tool)
– Click on one point/vertex
– you should be able to move all the point together

Resize or Rotate a mask

– Select the layer on your timeline
– Press M (Shortcut to access the mask properties)
– Select Mask Path
– Double click on any point/vertex
– Move your cursor on the corner of the layer you should see the resize cursor which should allow you to scale.
– Press Shift if you want to scale proportionally.

If you move you cursor further more on the corner you should also see the rotate cursor which will allow you to rotate your mask.

How to use the Mask Modes in After Effects?

After Effects have 7 mask modes options click on the drop down you will see: None, Add, Subtract, Intersect, Lighten, Darken, Difference
After Effects Mask Modes options

See below what After Effects mask mode options are doing visually

mask mode add  mask mode subtract
mask mode intersect After effects mask difference mode

How to create a Shape Layer Mask in After Effects?

– Make sure no layer is selected on your timeline
– Top tools bar, click on the shape tool icon after effects shape tool icon (Shortcut: Press Q)
– Draw a rectangle
– Make sure your rectangle shape layer is above the image/video layer.
– Select your image/video and change track matte to Alpha

See example below or Download the After Effects Shape Layer mask project.

AE shape layer mask demo

Tips: Once you have created you shape layer mask you can quickly change the fill color and stroke color
– Just hold ALT then Click on the fill color or the stroke color, demo below

After Effects Shape layer mask shortcut

Draw with the Shape Tool in After Effects

The Shape tools  after effects shape tool icon (Shortcut:Press Q) allows you to create a mask or shape layer with different path such as
Mask Shape Tool– Rectangle (default shape)
– Rounded Rectangle Tools
– Ellipse Tool
– Polygon Tool
– Star Tools

Trace Mask with the Pen Tool in After Effects

If you need to create a custom mask shape the Pen Tool  (shortcut: Press G) will allow you to trace a path with vertex points.

When you choose to trace with the pen tool you also have access to several tools to edit your path such as:
After Effects Pen Tools Options– Pen Tool
– Add vertex Tool
– Delete Vertex Tool
– Convert Vertext Tool
– Mask Feather Tool

Quick demo of the pen tool below

mask with pen tool

TIPS:
Check mark RotoBezier it will make automatic smooth curved path for you.
Uncheck Rotobezier to have more control on your points vertices with Bezier handles
RotoBezier

How to edit and modify a shape layer mask ?

When you create default shape layer mask such as rectangle, eclipse, polygon you will quickly see that you can’t move the point/vertex.
In order to modify and control the point/vertex you will need to right click on the path and select Convert to Bezier path
Convert shape layer mask to bezier path

To modify or animate your shape layer mask there are handy edit tools (Merge Paths,Offset Paths,Pucker & Bloat,Repeater…), you can find them under the Add menu (screenshot below), I suggest you test each tool to familiarize yourself, if you need specific detail on each add menu tools you can also visit this page on the adobe website.
Shape layer Mask Add Menu Tools

If you haven’t found what you need on this page please post your questions down below and I will improve this post, you can also find more on After Effects masks on the Adobe website here.