This is one of the limitations of 2D motion blur. When applied to full frames RSMB usually works best with relatively subtle motion blur. However to avoid hard edges when applying heavy motion blur, you can separate your objects via render layers or with Cryptomattes. Then apply motion blur to each render layer individually. Allowing you to fine tune your composite as needed. If required, piping your render layer's alpha output into the RMSB Motion Vector AOV node's alpha input, will also give you the motion vector pass of your object(s) on a transparent background.

The exported RSMB motion vector AOV will work with any third party tool supporting Reel Smart Motion Blur. Including: After Effects, Blackmagic Fusion, Nuke, DaVinci Resolve, Media Composer, Baselight, Final Cut Pro, Premiere, Natron, Apple Motion etc. The download files also include an installable macro .settings file for a node group you can use directly within in Blackmagic Fusion.

Use the range slider to adjust the range of values in your motion vector AOV. Higher values are needed for scenes with fast-moving objects. Lower values are more suitable for scenes with objects moving more slowly. The default value of 20 is usually appropriate for the majority of scenes. When inspecting your RMSB Motion Vector AOV pass in the compositor, you'll need to set the slider to a value that returns no black pixels i.e. the image should appear as tones of red and green only. Failing to set the sensitivity slider correctly may lead to artefacts in RSMB so be sure to pay attention to this setting.

For maximum flexibility and to save headaches with having to re-render your sequence, in the case of setting this slider incorrectly. We suggest saving the regular Cycles motion vector pass to disk, along with the rest of your passes/AOVs. That way should you need to adjust the RSMB Motion Vector AOVs node to get the correct output after rendering, you will still have the ability to do so.

• Are you rendering your Motion Vector AOVs to 32 bit EXR?
  Due to the data they contain, motion vectors require you to render to a linear format like EXR, ideally in full float (32 bit colour) for the highest accuracy. If you're rendering to a non-linear format such as JPEG or PNG, RSMB won't be able to interpret the data in the image correctly and you'll end up with incorrect motion blur and visual artefacts.
  
  
• Is there any black visible in your Motion Vector AOV?
  When inspecting your Motion Vector AOV pass in the compositor, you'll need to set the range slider to a value that returns no black pixels i.e. the image should appear as tones of red and green only. Failing to set the sensitivity slider correctly may lead to artefacts in RSMB, so be sure to pay attention to this setting and do some tests with frames from the start, middle and end of your sequence.
  
  For maximum flexibility and to save headaches with having to re-render your sequence, in the case of setting this slider incorrectly. We suggest saving the regular Cycles motion vector pass to disk, along with the rest of your passes/AOVs. That way should you need to adjust the RSMB Motion Vectors node to get the correct output after rendering, you will still have the ability to do so.
  

To avoid hard edges when applying heavy motion blur with RSMB, it can be beneficial to create a Motion Vector AOV pass per object or render layer. If this is required, the alpha input on the node group will allow you to render object(s) on a transparent background. Connect the alpha channel of your render layer to the node group's alpha input, or the output of a black and white mask created with another method of your choice such as cryptomatte etc. If you use the alpha channel of your render layer however. Be sure to check the Transparent checkbox under Render Properties > Film to ensure transparency is enabled in your scene first.

No, the standard motion vector AOV created by Cycles isn't compatible with RSMB.

You can append the node group from the supplied product file to any Blender project using the usual file > append method. The node group will then be available in the Compositor's add > group menu. Be sure to enable the Vector pass under the View Layer Properties tab though first.  If you plan to use the node group regularly, we highly recommend purchasing the Node Kit addon and adding the node group to your library for easy access.