We have that set up and it has been working for a very long time. In our Reconciling services, it is enabled in the Reconcile Error task. For the Posting service, it is enabled in the Reconcile Error task, and for the Post Error task. I have tested it in the Validation Error task, but not implemented in production. This feature just seems to work
Outlook is not running on the GDBM server--GDBM itself creates the message and sends it off to the value specified in the MailServer parameter.
Very basic, but can the GDBM server communicate to the SMTP server? For example, in a web browser on GDBM server, can you hit https://mail.mydomain.com:443 ... or maybe http://mail.mydomain.com:25? If so, mail.mydomain.com should work as your MailServer.
That's right. The SMTP server needs to be actually running the SMTP email server software. So it would need to be the actual server host that your outlook client connects to for getting the email. So you can check your outlook settings to find the server host that you use for sending your emails and is normally something like Matt said: "mail.yourdomain.com". You can normally just get this info from your network administrator that set up the email clients for everyone.
You may also have to check with your email administrator. Assuming you're dealing with Exchange and you're set up has typical security settings, you need to register the IP address of the machine running GDBM with Exchange. Otherwise, Exchange will ignore any attempt by that machine to send an email via SMTP. We had to do this with Responder.