Since Apple switched to using launchd instead of initd to start system services such as sshd, the way to configure the port on which the service is listening is also different from the regular editing of sshd.config. (No matter what I tried before, I couldn't get it to work. Now I know why)
It involves changing a bit in the launchd configuration and in the etc/services file.
More info is in this article.
In my case, where I wanted to add a second port where sshd should listen on, so as the article suggested I added a second (ssh2) service config file for launchd and also added the additional ssh2 port in the services file. After telling launchd to reload the config, it worked!