And I’m pretty sure this isn’t just a case of the correct characters lurking in the DB and being downsampled on their way to me, as I have WordPress configured to use UTF-8, the site’s head contains a meta that declares UTF-8, and a peek at the HTTP response headers shows that I’m serving UTF-8.  Of course, I’m not really expert at this, so it’s possible that I’ve misunderstood or misinterpreted, well, just about anything.  To be honest, I find it deeply objectionable that this kind of stuff is still a problem here on the eve of 2010, and in general, enduring the effluvia of erroneous encoding makes my temples throb in a distinctly unhealthy fashion.

Anyway.  Moving on.

I found a search-and-replace plugin—ironically enough, one written by a person whose name contains a character that would currently be corrupted in my database—that lets me fix the errors I know about, one at a time.  But it’s a sure bet there are going to be tons of these things littered all over the place and I’m not likely to find them all, let alone be able to fix them all by hand, one find-and-replace at a time.

What I need is a WordPress plugin or something that will find the erroneous character strings in various fields and turn them back into good old UTF-8.  Failing that, I need a good table that shows the ISO-8859-1 equivalents of as many UTF-8 characters as possible, or else a way to generate that table for myself.  With that table in hand, I at least have a chance of writing a plugin to go through and undo the mess.  I might even have it monitor the DB to see if it happens again, and give me a big “Clean up!” button if it does.

So: anyone got some pointers they could share, information that might help, even code that might make the whole thing go away?

Also, there is an “enter debug mode” link at the bottom of the administrative panel for the plugin.  It’s very cleverly matched with an “exit debug mode” link when you’re in debug mode.  These links do just what they sound like they should do.  Debug mode itself, introduced in the previous beta, is unchanged (except maybe cosmetically).

In case anyone’s interested in seeing how I use the text-replacement strings on  meyerweb, here’s what I have in that textarea, formatted slightly for readability:

<div class="panel">
<h4>Recently Tweeted</h4>
<p class="more">
<a href="http://twitter.com/%%USER_SCREEN_NAME%%">see more</a>
</p>
<p>
%%TEXT%% <small>–tweeted %%CREATED_AT%%</small>
</p>
</div>

There were some reports of incompatibility between this plugin and early WordPress 2.7 betas.  Word is it’s working fine with the latest beta.  I probably won’t fix any incompatibilities until 2.7 final ships, but if anyone spots something they absolutely know will be a problem in 2.7 final, let me know.  Thanks!

1. All the files are in the mw_latest_tweet directory now, instead of having the plugin PHP outside of that directory like 1.0 did.  Yeah, I know, that should’ve been the case all along.  Sorry!  Learning on the job here.

   If you’re upgrading from 1.0, you should probably delete the 1.0 file and directory outright before uploading the 1.1b1 directory.  Alternatively, you should be able to upload 1.1b1, deactivate 1.0, activate 1.1b1, and then delete just the 1.0 PHP file.  I haven’t tried that, so I don’t know if it will actually work, but it seems like it should.

2. URLs within a tweet are turned into hyperlinks for easy clickin’.  To go with this new feature, there’s a new option on the settings page to either shorten displayed URLs, like twitter.com does, or to not shorten them.  The default is to shorten, which means any URL 29 or more characters long gets shortened to 27 characters and gains a trailing ellipsis.  Again, like Twitter does it—although I used an ellipsis entity and not three periods.

   Note that if you upgrade from 1.0 to 1.1b1, this setting may default to “No” instead of “Yes”.  I’m not sure why, but it’s a pretty low-priority item right now.

3. On a related note, @names are autolinked as well.  I’m using the pattern [A-Za-z0-9_] since that’s what Twitter says are valid characters for a username even though if you type in a grawlix on the signup page it will tell you, in nice bold green letters, that it’s available.

4. If you want to see everything the plugin has cached, append &debug to the end of the plugin’s setting page URL and hit return.  You’ll get the settings page with a dump of the cached data at the end.  This is clumsy and will be much less so before 1.1 final.  I’m thinking click a link, enter debug mode.  Probably won’t go all AJAXy, though you never know.

So that’s the state of things.  Let me know if anything breaks.

So I have meyerweb’s WordPress install all subversion-managed.  The only problem is that there are three core files I’ve had to hack (reasons upon request) and that makes updating really icky.  I fire off…

svn sw http://svn.automattic.com/wordpress/tags/2.6.1/ .

…(where 2.6.1 is replaced with whatever the latest version is) and it updates everything.  For my custom-altered files, though, I get diff files and a .mine file that has my old copy and then a copy that’s littered with diff markers, which cause PHP error-crashes, which takes down the site.  At least until I go in and copy the .mine files over the diffed-up files.

So: how do I do some kind of local checkin of the altered files so that I don’t attempt to post them back to the WordPress codebase (these are very specialized hacks) but future WordPress updates don’t break my site?  For extra ideal points, it would be great if those files were updated with my changes merged into the files.  If it helps, the files thus affected are /wp-blog-header.php, /wp-includes/classes.php, and /wp-admin/edit-form-advanced.php.  Thanks for any help!

I have another plugin request, but in this case I’m looking for help in modifying something I’ve already done.  Or half-done, maybe.

I don’t know about you, but I get a lot of comment spam.  As I type this sentence, Akismet has stopped 837,806 spam attempts in total.  A false positive makes it past Akismet and my other defenses to land in the moderation queue about once every four days, on average.

Some of those false positives are really, really, really easy to spot, and they get marked as spam in order to help improve the recognition algorithms.  Others are hard to evaluate just by looking at the comment.  Many are trackbacks from sites in langauges I can’t read, and others that I can read look legit enough.  In such cases, I usually go visit the author’s URL to see if it looks spammy or not.

Now, the way I used to do this was to right-click on the blog link, copy the URL of the target, open a new browser tab, and paste the URL into the address bar.  Why?  To prevent my WP admin URL from landing in the referer logs of a potentially unscrupulous site owner.  But sometimes I forget to do all that, and just click.  I figured, well, why not stop fighting the tendency to just click and write a plugin that routes all outbound links through a redirect service?

So I did.  You can grab it for yourself if you want, but if you do, understand that it’s pretty clunky right now.  Which is the part I’d like help fixing.

The heart of the plugin is simplicity itself:

if (is_admin_page()) {
	add_filter('get_comment_author_url','_mw_obscurify',5);
}

function _mw_obscurify($url) {
	if ($url) return 'http://google.com/url?q=' . $url;
}

There’s a little more to it than that (specifically, the routine is_admin_page(), which I got from someone else’s plugin and wish now I could remember whose it was) but that’s the core.  So any time the URL of a comment author is fetched, it’s prepended to turn it into a Google redirect.

That’s true for both href values and displayed URLs, though, which is the clunky part.  The end result is that on comments from the aforementioned mighty Hamish, for example, I get the following markup on the “Comments” page:

<a href="http://google.com/url?q=http://hamstu.com">

http://google.com/url?q=http://hamstu.com</a>

What I’d very much prefer is:

<a href="http://google.com/url?q=http://hamstu.com">

http://hamstu.com</a>

Or even:

<a href="http://google.com/url?q=http://hamstu.com">
hamstu.com</a>

So what I’d like to know is if there’s any way to make that happen short of rewriting and replacing get_comment_author_url, which I’d prefer not to do since it could change in future versions of WordPress and I’m not particularly interested in turning a basic plugin into a continuing maintenance headache.  I mean, I will if absolutely necessary, but I’d like to find a better way if there is one.  Thus the request for help.

Also, are there better redirect strategies than using Google the way I have?  It’s very slightly annoying that I have to click through the Google redirect page, and though I absolutely understand why they do that, I’d love to find an automatic redirect that wouldn’t expose my referer to the target site.  Anyone know of one, or have a related sharp idea?

I need a WordPress plugin that won’t let me publish a post until I’ve filled in the excerpt field. Anyone got one?

To which I got a whole lot of responses saying, in effect, “Oooo!  Good idea!  I need that too!  Let me know when you find one!”  Some of them came from people running fairly high-profile blogs.  The need clearly exists.  A couple of responses were of the “I could do that!” variety, so I thought I’d post here so as to describe how I think it ought to work from the user’s perspective, and then we can hash things out in comments and someone can code it up and make everyone happy.

So really what I want is, when I push the “Publish” button in WordPress, the plugin checks to see if there’s an excerpt.  If not, one of two things happens:

1. The plugin throws up a warning dialog telling me that I’m about to post without an excerpt (again).  If I say “Okay”, it goes ahead with publishing.  If I say “Cancel”, it returns me right back to where I was, which is the “Write Post” page, with all the data intact and unaltered.

2. The plugin returns me to the “Write Post” page with all data intact and unaltered, and puts an error box at the top of the page telling me I forgot to write an excerpt (again) and that it won’t let me publish until I fix the problem.

One or the other.  I think I like #1 a little better, but I’d be good either way.  I’m open to other approaches as well, but I don’t think the plugin should rely on JavaScript, as that means leaving out people who don’t enable JavaScript or post from JS-incapable devices.

I would do this myself, but I’m a little wary of the “return to the page with all data intact and unaltered” bit, which I would imagine is pretty easy to mess up.  Thus I’m putting it up here as a semi-Lazyweb post so that someone else, someone with more experience with WordPress and plugin authoring, can do it right and quickly.

Okay, who’s on it?

Okay, second’s up!

MW Adminimize (currently v0.5)

This plugin pares down the page-topping links interface that shows up on every WordPress administration page.  I got tired of seeing my blog name in Big Huge Letters all the time, and I kept forgetting that the “Dashboard” link was hidden up in that black bar.  So I fiddled with some arrays and wrote some CSS, and got something that chews up less screen real estate while also being easier for me to use.  Maybe you’ll feel the same, in which case, enjoy the plugin.

In addition to that, I pared down some of the elements of the “Write Post” page so that they took up a bit less space, and compressed the stuff around the post title’s input field.  Those styles are marginally less stable and may cause weirdness for you.  If so, sorry.  Let me know what goes south and I’ll try to get it fixed in a future version.

Oh, and I also included a CSS rule that makes the background shading of moderated comments more obvious, though not in my usual eye-scarring fashion.  It’s just a little darker and notably redder.  That doesn’t have much of anything to do with minimizing the admin UI, but I thought it was a huge improvement.  I might take it out in a future version if the crowd shouts otherwise.

MW Latest Tweet (currently v1.0)

This one does what you’d probably guess: shows the latest tweet.  Why not just use something like Twitter Tools?  Actually, I tweeted about that.  A further survey of the options available turned up nothing that was quite what I wanted.  So of course I started hacking away.

The basic mechanism is that any time a page that calls this plugin is loaded, it either returns the last-cached tweet or asks Twitter for the latest tweets so it can cache the latest of them.  If Twitter fails to respond (I know, what are the odds?), you still get the cached tweet.  This is similar to SimpleTwitter, from which I took much inspiration, but is more advanced in a number of ways.  Among the options I wanted to have and thus created was having “human time” intervals as an option, and also the ability to filter out @replies (because I personally wanted to display my latest broadcast tweet, not my latest reply).  While I was at it, I decided I’d throw in the ability to define your own XHTML output, so that if the plugin failed there wouldn’t be partial or mangled markup shoved into your pages.  Also, since I cache the whole of the latest tweet, you have all kinds of stuff you can output should you so desire.  It’s all documented on the plugin’s page.

This is not a widget, because I’ve never written one and didn’t have time to learn (due to all the time I was burning just trying to write the plugin).  You have to edit your template/web page PHP in order to call it.  That’s just the kind of DIY guy I fly.  Er, I am.

Also, I’m a wee tiny bit proud of how I styled the labels on the plugin’s Settings page.  I’ll probably write that up in a separate post.

That’s about it.  If anything goes wrong or is unclear, please do let me know in the comments.  Share and enjoy!

At the most basic level, all I really want is for the plugin to die() with a message telling the commenter that their comment has been Akismetted and that they should contact the site administrator about it if they’re legitimate.  If instead of dying it gave that message upon returning them to the post on which they commented, the way “Your message is awaiting moderation” works, that would be extra-cool.  Either way, it obviously only fires if the comment has been marked spam.

I’m guessing this is a pretty simple plugin to write.  I did attempt to write it myself, but was spectacularly unsuccessful in the attempt.  The really frustrating part is that I think I got close to making it work, but, in the end, failed badly.  Then I asked for plugin help on IRC, but nobody noticed.  So I’m throwing the request out to the audience here to see what y’all can do.  Thanks to anyone who can help!

Update 5 Feb 07: Mark Jaquith has come through with Antispam Collateral Condolences, a plugin that does exactly what I requested.  It can put up the warning either as a JavaScript alert box, or a message embedded in the page upon returning them to the post on which they commented.  Totally, completely awesome.  You rock, Mark!

In the end, a post from Alex King showed me the way to a “fix”.  I put that word in quotes because this is really more of a hack.  No, actually, it’s just gross, but it works.  In order to get WP to stop throwing inappropriate 404s, I commented out lines 35 through 135 of classes.php in the WordPress core.  Here’s the beginning and end of the block I commented out:

if (! empty($rewrite)) {
   // If we match a rewrite rule, this will be cleared.
   $error = '404';

[...a ton of redacted lines...]

      $this->did_permalink = false;
   }
}

There you go: with one hundred lines of code commented out, my feeds and archive page work again.  This is not a good idea, but it’s all I could figure out to do.  Obviously, having modified a core file, I’m taking a chance that this won’t cause other problems.  So far I haven’t seen any, but you never know.  I’m also making upgrades much harder on myself, given that the new core file won’t have my modifications.

Similarly, I commented out the mod_rewrite.c block WP added to the end of my .htaccess file, so that I can use my own rewrites and not invoke WP’s rewriting.  With every upgrade, I’ll have to check that file to make sure the block didn’t get added back.

I’m mostly posting this as a reference for anyone who might run into similar problems with their WP 2.x installation.  It’s my hope that a future version of WP will offer some kind of hook or option that will let me easily disable WP’s rewriting and error handling code, since I already have a web server that’s configured to do those things for me.  I don’t mind if it’s a hook that I can address via a plugin as opposed to an option in the preferences, but I’d like to have it.  It would make WP a little more attractive to power users as well, since they’d know that they could do their own rewriting and error handling without having to fight with WP’s internals.

Of course, I only got to that point after commenting out the IfModule mod_rewrite.c stuff WP apparently tacked onto the end of my .htaccess file.  Before that, the above-referenced feed URLs just brought up the home page of the site while preserving the feed URL in the address bar.  Keen.

Exactly the same progression of trouble has applied to the Distractions feed, too.  At first the server (or WP) was filling in the home page, and now it returns the feed along with a 404 response.

I’ve also had the same problems with the main page of the post archive—at first, it redirected to the home page.  Now it just returns a 404, although unlike the feeds, it doesn’t also create a full page to go with the 404.  At least, I don’t think it does.  I can’t see it, anyway.  A shame, too, that it does its own thing rather than use the 404 page I already defined in the .htaccess file.

Anyone have any guesses at all as to what’s going wrong, and how I can get WP to stop throwing around inappropriate 404s?  I assume this has something to do with my .htaccess file’s Rewrite rules—which worked just fine in WP 1.5.2, by the way—and the new error handling routines.  Or maybe it’s just the new error routines.

Oh, and while we’re at it, what’s the story on the “new tabbed editor” that’s supposed to let you “switch between WYSIWYG and code editing instantly while writing a post” that was mentioned in the WP 2.1 announcement?  Because I’ll be switched if I can find it.

Ah, the joys of upgrading…

Addendum: anyone know of a quick and simple way to just disable WordPress’ internal rewrite code, so I can go back to using my .htaccess file?  That might solve the problem just as well as trying to hack my way around the problem with WP, and a lot more simply.

For those who joined the party in the long silence since RC4, Gatekeeper is a WordPress plugin that lets you manage a series of challenge/response pairs.  The default challenge is “What color is an orange?” (correct response: “orange”), though you should definitely disable that one and add your own.  This helps stymie spambots, though of course it is easily defeated by a manual spammer—and they do exist—and it can do nothing to stop trackback spam.  I actually stopped using Gatekeeper on meyerweb when I installed Akismet, which may be good enough for most people.  For those who can’t or won’t run Akismet, though, Gatekeeper is a decent alternative.

Gatekeeper is technically a CAPTCHA, but it is a fully accessible CAPTCHA, as it uses no images.  It’s also highly configurable, allowing you to add as many challenges as you like and then rotating between them randomly.  I know of a few sites that are quite happy with Gatekeeper, and recently caught wind of a Django implementation of the same concept.

So it’s there and ready for use by those who are interested.  If I haven’t heard about any bugs within the next month or so, I’ll strip off the RC designation and go with 1.5 final.  And about time, too.

Note to WordPress 2.0.x users: I have no idea if WP-Gatekeeper 1.5 will work in WP2.  It may.  Then again, it may not.  I’d be interested to know either way.

I don’t pretend to understand all the details of Akismet’s workings, although I have a fairly good idea of how it works.  I have some concerns, mostly in that it seems like spammers could poison the well by injecting tons of false “not spam” data into the service in order to get their messages through.  I also worry about attacks on the service itself.

Furthermore, I have to say it’s a bit frustrating that you have to have a wordpress.com API key, which means you have to have a wordpress.com account, which means it’s not a one-stop plug-and-play solution.  (Especially since getting an account is, currently, an invitation-only sort of thing.)  On the other hand, having to have an account probably confers some control advantages—if an account is found to be consistently marking things as “not spam” when everyone else is marking them spam, it can be kicked out of the service.

Some have raised privacy concerns because every comment submitted to your site gets analyzed by the Akismet service.  This doesn’t bother me, but it might some.

Overall, I’ve been pretty happy with Akismet.  It has let through less spam than Gatekeeper did in the weeks before I disabled it and all my other anti-spam measures to test out Akismet.  You’d think a Gatekeeper setup wouldn’t let anything through, but you’d be wrong; I assume there was a hole in my PHP.  Akismet may not be the end-all solution—after all, if it becomes effective enough, the spammers will have major incentives to defeat it, and will most likely find ways to do so—but it seems to be working very well for now.

The point of this being to say that I’ve updated Gatekeeper to return a more helpful error message, and also believe some changes I made should make gatekeeper less random.  If I’m wrong and you get an error, please use the e-mail link in the error message to send the error to me.  If I’m right and you get no error, then no worries.

If I get a lot of error reports, I’ll hobble or disable Gatekeeper, but I hope it doesn’t come to that.  We’ll see.  In the meantime, my sincere apologies to anyone who got bitten by this bug, and I hope things will be better from here on out.

The auto-addition of the challenge to forms that don’t contain the pose_challenge call is still broken.  ‘Jim’ has more information and some examples of forms that fail to get an auto-added comment if you’d like to take a crack at solving that particular puzzle.

What this means is that Gatekeeper is, as it always was meant to be, a way to prevent comment-form spambots from succeeding.  Trackbots will now get through unless you take other steps, like disabling trackbacks or running another spam filter or something.  I’d actually like to see WordPress split tracks/pings apart from comments, and let you set their “always moderated” flags separately.  Thus you could set things up so all tracks and pings are moderated, but comment-form comments are not.  That would work great for me.  Maybe not so well for others, though.

Unfortunately, rc3 still has that problem where it doesn’t always automatically add a challenge to your comment forms, though you can still get the challenge by manually adding gatekeeper_pose_challenge to the comment forms in your theme.  My grep-fu (or maybe it’s my PHP-fu; or, hell, both) is weak; I can’t figure out why the routine fails.  Anyway, head on over to the Gatekeeper page if you’re interested, and especially if you can figure out why the auto-challenge routine is failing.  Thanks.