Got around to setting up IPSec between my OSX and Linux server, it worked just fine with PSK (pre-shared key), but failed when using certificates, the error on the server side was:

generating ID_PROT response 0 [ KE No CERTREQ NAT-D NAT-D ]

I enabled debugging of racoon on OSX (add the 2 lines to file /etc/racoon/racoon.conf)

log debug;
path logfile "/var/log/racoon.log”;

and got:

Jul  3 10:33:48  racoon[19904] : created CERT payload
Jul  3 10:33:48  racoon[19904] : use ID type of DER_ASN1_DN09000000 3032310b 30090603 55040613 02504c31 0d300b06 0355040a 13044b75
62613114 30120603 55040313 0b4b7562 61206950 686f6e65
Jul  3 10:33:48  racoon[19904] : hmac(hmac_sha2_256)
Jul  3 10:33:48  racoon[19904] : error -25308 errSecInteractionNotAllowed.
Jul  3 10:33:48  racoon[19904] : failed to sign.
Jul  3 10:33:48  racoon[19904] : failed to get sign
Jul  3 10:33:48  racoon[19904] : failed to allocate send buffer
Jul  3 10:33:48  racoon[19904] : IKE Packet: transmit failed. (Initiator, Main-Mode Message 5).
Jul  3 10:33:48  racoon[19904] : sending vpn_control ike failed message - code=65535  from=local.
Jul  3 10:33:48  racoon[19904] : failed to process packet.
Jul  3 10:33:48  racoon[19904] : Phase 1 negotiation failed.

The error seemed to indicate issues signing the message, so I started poking around, and realised that racoon may simply have no permissions to my private key, the fix was easy:

in Keychain Access, find the private key portion of your certificate, double click on the private key and in the “Access Control” tab, add a new application to the permission list, you may need to press Command+Shift+G to open “go to location”, enter “/usr/sbin” and then find a “racoon” binary.

Save and you should be good to go.