SharePoint 2010 Error 8313 Dead SearchService.svc

Search is dead on your SP2010 farm.

You see this error in the ULS logs:

CoreResultsWebPart::OnInit: Exception initializing: System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Office.Server.Search.WebControls.CoreResultsWebPart.SetPropertiesOnQueryReader() at Microsoft.Office.Server.Search.WebControls.CoreResultsWebPart.OnInit(EventArgs e)

You also see this error in your server application logs:

SharePoint Web Services Round Robin Service Load Balancer Event: EndpointFailure
Process Name: OWSTIMER
Process ID: 7596
AppDomain Name: DefaultDomain

AppDomain ID: 1
Service Application Uri: urn:schemas-microsoft-com:sharepoint:service:2313ae1967844030a54407c13c36d972#authority=urn:uuid:3f2c33a4777749139422ac1249fc9288&authority=https://server:32844/Topology/topology.svc
Active Endpoints: 1
Failed Endpoints:1
Affected Endpoint: http://server:32843/2313ae1967844030a54407c13c36d972/SearchService.svc

You have tried everything that every other blog post, forum answer, KB article has told you to do. You've used your insanely good SharePoint debugging skills, you have rebuilt your search service application (2x) and for some reason this error just won't go away.

I have no idea what causes this, but I know how to fix it.

To know for sure that you are having the same problem as me, find the virtual application in IIS Manager under "SharePoint Web Services" that has the same ID as referenced in BOLD above in the windows application logs. (If you aren't actually doing round robin with this service, it may be on one server and not the other, so check both). See the connection in the pic below to the error above?

Click advanced settings and find the application pool it's running under. Mine was a bunch of crazy letters and numbers, like 71283482842042sdf203402. Go to the application pool under "application pools" and you'll see it is stopped. Am I right so far? Well the problem you are having seems to be an issue with the managed account that app pool is running under not working anymore for some reason. Here's how I fixed it:

1. Logged into AD users/computers and manually reset the password to something I know.
2. Logged into WFE and went back to IIS Manager and reset application pool credentials to use new password.
3. Started app pool and made sure it stayed up
4. Logged into WFE server and ran SharePoint powershell command: Set-SPManagedAccount -Identity domain\searchapppoolAccount -UseExistingPassword
5. Updated the services MMC with this new password and restarted the SharePoint Server Search 14 Service (on all machines in the farm)
6. Now navigate on your WFE server to the URL that I had bolded above and instead of seeing a 503 error, you should now see a nice service WSDL page: http://server:32843/2313ae1967844030a54407c13c36d972/SearchService.svc

No idea what just happened, but it worked. Now, beer.