[Ads-l] How to search new ADS-L archive?

[ADSGarson O'Toole]

Perhaps the robots.txt file associated with the website has been
changed intentionally or unintentionally. The robots.txt includes
guidance messages that are read by well-behaved webcrawling robots.
The messages can specify parts of the directory hierarchy that
webcrawlers (like Googles) are supposed to refrain from traversing and
indexing.

http://www.robotstxt.org/robotstxt.html

[Begin excerpt]
About /robots.txt

Web site owners use the /robots.txt file to give instructions about
their site to web robots; this is called The Robots Exclusion
Protocol.
[End excerpt]


On Fri, Jul 31, 2015 at 11:51 PM, Ben Zimmer <bgzimmer at gmail.com> wrote:
> ---------------------- Information from the mail header -----------------------
> Sender:       American Dialect Society <ADS-L at LISTSERV.UGA.EDU>
> Poster:       Ben Zimmer <bgzimmer at GMAIL.COM>
> Subject:      Re: How to search new ADS-L archive?
> -------------------------------------------------------------------------------
>
> On Fri, Jul 31, 2015 at 10:10 PM, Neal Whitman wrote:
>>
>> Is it because the website isn't mobile-friendly? I remember hearing Google
>> was going to implement a policy like this a month or two ago.
>
> No, that change only affects search result rankings (privileging
> mobile-friendly sites for mobile search). And from what Grant says,
> it's a change on the Linguist List side that has led Google to stop
> indexing the archives. I see that the main Linguist List archives are
> still getting indexed, though.
>
>>On Fri, Jul 31, 2015 at 2:54 PM, Ben Zimmer wrote:
>>>
>>> On Fri, Jul 31, 2015 at 2:22 PM, Joel Berson wrote:
>>>>
>>>> Grant, I think Ben Zimmer did something (a specialized Google search?), but
>>>>  I've lost track of what.
>>>
>>> Just look down below in the thread -- the post that Grant was replying
>>> to has the link: http://goo.gl/gLGhEs
>>>
>>> But the custom search I created is really just a Google site search,
>>> so the indexing problem that Grant is talking about would affect that
>>> too.
>>>
>>> From the looks of things, the Google indexing stopped in mid-April.
>>>
>>>>
>>>>  From: Grant Barrett <gbarrett at WORLDNEWYORK.ORG>
>>>>  To: ADS-L at LISTSERV.UGA.EDU=20
>>>>  Sent: Friday, July 31, 2015 12:51 PM
>>>>  Subject: Re: [ADS-L] How to search new ADS-L archive?
>>>>
>>> FYI, it looks like the folks at LinguistList have stopped letting
>>> Google index the archive. I'll see if I can get that reinstated, but
>>> it means that your searches of the ADS-L archive will not include
>>> recent material.
>
> ------------------------------------------------------------
> The American Dialect Society - http://www.americandialect.org

------------------------------------------------------------
The American Dialect Society - http://www.americandialect.org