The downside is that War Mode will make you vulnerable to other players, which means you might be killed more often. Whether it’s worthwhile will depend on the zones you’re visiting and how active the other faction is on your server cluster. Heavily populated realms usually mean more players on both sides, and more conflict, but on quieter realms, you may rarely run across an enemy.
Chilled to the Bone, Dampened Magic, Kleptomania, Netherwind Armor, Prismatic Cloak, and Temporal Shield have limited use in PvE content in terms of increasing your damage in the open world, but they can be situationally useful. Notably Temporal Shield can be utilized for, functionally, an extra 6 seconds of combat time against large elite mobs.