Can I set the captcha to 5 characters?

No, 5 characters is below the allowed threshold. Setting the field to 5 or fewer characters will trigger a save error and prevent your settings from updating.

The minimum requirement is 6 characters or higher.