ifconfig

sudo ifconfig fxp0 delete 123.123.123.123

Just to be safe you can remove the route as well.

sudo route delete 123.123.123.123

Also you need to make sure the ip address entry is removed from
/etc/start_if.fxp0 of /etc/ifconfig.fxp0

ip (for computers without ifconfig)

sudo ip addr del 123.123.123.123/24 dev fxp0

removes route and ip

Also you need to make sure the ip address entry is removed from

/etc/sysconfig/network-scripts/ifcfg-fxp0