This can occur if you are trying to specify a new filter type without having the coefficient vector populated with a defined value. This is a known issue with the FIR Compiler 5.0 as implemented in System Generator for DSP.
To work around this issue, ensure that your coefficient vector variable is defined in the workspace before opening the FIR Compiler GUI and making changes to the filter type. If you have already done this and cannot enable the field, you can select either interpolation or decimation (whichever one is currently not selected) and then change back to the desired value. This should enable the appropriate "Rate Value" field in the GUI.
This issue is scheduled to be resolved in a future release.