Do I need to change the file extension for my cert_key_pem.txt file to .pem for it to be recognized as PEM format?

No, the issued cert_key_pem.txt file you download from PayPal is ready for immediate use. You won't need to manually change the file extension for it.

Note: This file contains both your public certificate and your private key. It should remain unaltered and be stored in a secure location at all times.

See also:
Creating and managing NVP/SOAP API credentials
Merchant API Certificate Credentials Upgrade
Secure your HTML payment buttons - Setup Certificates

More ways we can help