RSS feeds and Google blog search

[cjsmith]

Paraphrasing some stuff in a friendslocked entry I found. Good info; I'll try to do it justice. Hat tip to runeshower.

Google is currently indexing the contents of LiveJournal blogs' RSS feeds.

RSS feeds don't (at the moment) contain a "block robots/spiders" setting. Google is working on how to respect this setting ANYWAY; they expect to have a fix within the next week.

RSS feeds contain only your public entries. If private or friendslocked content shows up on Google Blog Search, it was indexed at a time when those entries were public. To request its removal you'll have to talk to Google about it.

You can't completely shut off your RSS feed, but you CAN shrink it down to contain only the titles of your entries. To do this, go to the LiveJournal console and enter the command set synlevel title.

_____________________________


I've done the "set synlevel title" command, and I have "block robots/spiders" checked. However, now that I'm thinking about this a bit, I'm half inclined to allow spiders and let Google index my posts. One of the reasons I flail about with LJ memories so much is that it's so danged hard to SEARCH. Google is certainly equipped to solve that problem! And hey, all public content was already public well before Google began caching it. Pretending there are folks who can't see it or won't find it is a recipe for getting badly hurt. All I'll have to do is make sure I don't put my e-mail address in my entries. I get enough spam.

Comments

[sunnydale47.livejournal.com]

Trying to crawl back after a long absence ....

Your final paragraph is very helpful, since I'm flailing in the same way for the same reason. I have bots and spiders blocked because I don't want a bunch of strangers showing up from all over the web -- but as my list of memories grows to where I need search engines just for them, maybe it's worth it. I'll never have enough links to my journal to get a PageRank above 827 or so anyway, so what's to worry about?!

If you unblock spiders, will Google crawl all the way back and index all the public entries? Or only the ones that are posted after the unblocking?

[eichin.livejournal.com]

ah, I was wondering why all of your entries showed up (title only, blank content) in my rss reader. They still serve as a reminder to check the friends page too. (One of these days I'll make my reader use an authenticating proxy; I'm pretty sure you see friends-locked posts in rss if you fetch it when logged in...)

Another thing to know about is that there are "live feeds" of all public postings in real time. That's probably not what google uses (since they have older postings too, though it might be a way to discover feeds to look at), but technorati and that sort of thing do. (I pulled that feed into an appliance at work for a couple of days, as a volume scaling test, but also to see if lj is "interestingly geographic". Lots of good "casual" writing...) I assume synlevel controls that too.

[anisoptera.livejournal.com]

For useful. Thank you very much. 98 is concerned about the issue of seeing parts of locked posts in a google search. I've pointed him at your post.

[98.livejournal.com]

Thanks much for the info. Where is it documented? I tried using the 'help' command at the console to get a list of properties in case there is anything else I might wish to set but no joy.

This RSS feed business definitely has me miffed although I am not certain how sensible being miffed is. I may decide that I am being like one of those daft geezers who throw a fit because someone linked to his publically accessible web page without his explicit permission. Or I may decide that I am justified in my upset and do something like friends-lock my LJ or just abandon it.

Meanwhile I think it is the being blindsided and my perception that LJ officialdom gives an insufficient damn.

Thanks again.

[vito-excalibur.livejournal.com]

Very useful! Thank you.