The baudrate of the UART port when loading uboot does not depend on the value from the environment variable "baudrate" and is determined only by CONFIG_BAUDRATE. The rate can only be changed in the local session using the "setenv" command. The next time you reset, uboot will take the rate from CONFIG_BAUDRATE. "config set" has no effect too.
Have to recompile to change rate.
It is unusual for a design to change its debug UART's baud rate dynamically. The majority of our designs use the 115200 baud rate. If your design needs a different baud rate, I would rebuild u-boot and change CONFIG_BAUDRATE and change the bootargs so the kernel also uses the new baudrate.