I was under the impression that XP had no limit set on the amount of system resources that could be used. However when I have a bunch of IE windows open (say 15) and maybe 5 or so other applications plus whatever is running in the background... as I am prone to do my system starts to poop itself. Switching to applications from the task bar may not work and if I open a new window from whatever application this may not open.

Closing an IE window or 3 and or a couple of applications generally fixes the problem so obviously something is reaching it's limit. Does anyone have any idea what could be causing this and is there any way of setting the limit of whatever it is to a higher amount? I have 512MB of RAM and the page file is nowhere near it's limit. 

"I was under the impression that XP had no limit set on the amount of system resources that could be used."

That is the wrong impression.  XP is limited to the amount of resources you have.  XP does, however, allocate resources better than its predecessors.  Chances are some application you have launched is peaking out your CPU's, maxing them out a 100% and freezing or slowing down your system.  Would suggest launching Task Manager (right click on the task bar and select Task Manager from the popup menu) and click on the Processes tab and see what is eating up your resources.

No CPU max out. To get the pc to the stage of not opening new windows etc I open Photoshop. Load up a file. Open a few more copies of this IE window and my taskbar stops functioning normally and photoshop won't open any more windows.

Sure the PC is pretty loaded but it's not out of memory, it's not having the CPU sucked dry. It's not thrashing madly on the page file. And it's not just this pc.