When you saw the error is your crawl log:
Access is denied. Verify that either the Default Content Access Account has access to this repository, or add a crawl rule to crawl this repository. If the repository being crawled is a SharePoint repository, verify that the account you are using has "Full Read" permissions on the SharePoint Web Application being crawled. ( HttpStatusCode Unauthorized The request failed with HTTP status 401: Unauthorized. )
Reason:
This error will occur, at the time of crawling the SharePoint application content. because, you may have provided a URL to access the SharePoint web application like http://kenportal.com.au(just an example). If you try http://kenportal.com.au from your server, you will find you cannot access it at all.
When the search engine tried to crawl the content using the kenportal.com.au URL, the server will reject the request. because, it will take the request as "LoopBackRequest".
Solution:
So you need to disable the LoopBackRequest in registry. To do this, just follow my instructions:
1. Go to command window and type regedit.exe
2. Once opened the registry editor, just navigate to "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa"
3. Right click on "Lsa" and create a new 32bit DWORD value
4. Rename it as "DisableLoopbackCheck" (Note: you cannot rename it actually. so, once created the DWORD value, directly paste/type as "DisableLoopbackCheck".
5. Then again modify the value of "DisableLoopbackCheck" as "1"
6. Close the registry editor.
7. Now start to crawl the content.