My game does the same thing after a few hours of gaming. Even to the point that "PCB" (Purge Cell Buffers) in the console doesn't do justice. Simply Save and Exit the game completely should resolve those issues. Bethesda games are notorious for Memory Leaks.
Now if you are saying after 5 mins of gaming this happens. There's obviously something wrong. A Mod conflict or something in the INI files.
Try using Fallout New Vegas Configator (Yes Confi-Gator) change somethings on there. Set the Threaded options on and the background loading on. Havok Threads to your CPU count e.g. Duo Core= 2, TriCore=3, i7=8, etc....
The Cell Buffers I would NOT change.
Sam