We’ve been having some seemingly random performance issues on some of sites and for a while I was quite stumped. Basically every so often the website would take between 16-20 seconds to load the page, although we were not able to predict when it would happen, as it occurred on different pages within the site.
My initial thought was that it was a network issue due to the fact that it was only being reported on some sites. Luckily the company subscribes to a external monitoring service so I was able to see the sites that it was happening on and how frequently. Once I had a look there I noticed that was happening on all the sites to varying degrees.
One of the servers was taken out of the DMZ to try and isolate the issue and it was only then that I noticed what was going on. With all the other sites shut down and only one or two w3wp.exe process to keep an eye on I was noticing that one of the processes was being shutdown and replaced with a new one. Now I know that this is the what happens when an app pool gets recycled but this shouldn’t have been happening as we had only 1 scenario enabled, the virtual memory limit setting:
As you can see this was set to 250Mb and process itself was never getting anywhere near that!! In fact even before I had thought it was a network problem I did actually check for this, however with so many w3wp.exe processes (~30) it was difficult to see them recycling, and because none of them were going near my limit I quick discounted this idea! So I went ahead and removed the setting and re-ran the tests. Low and behold it fixed the issue. Now the reason I had set it in the first place was due to getting System.OutOfMemory exceptions if too many bots were walking the site at one time. This happens due to the fact that we use the XsltCompiledTransform to clean up content coming from our Homebrew CMS. This generates a lot of assemblies and hence when to much content is accessed too quickly it runs out of memory. Initially I was wondering if there was a bug in IIS6 but it turns out that memory can be consumed directly from the swap disk completely bypassing the physical RAM and hence why the w3wp.exe never got anywhere near my limit.
Anyway in the end the fix was quite simple, just use the physical memory setting in IIS. 🙂