Comments on: Correcting Corrupted Characters

By: Eran Galperin

Eran Galperin — Thu, 04 Feb 2010 17:54:15 +0000

I’m not sure if this is still relevant, but since I didn’t see any mention of this in the other comments, and going by character set details you posted, the issue is probably in the connection character set / collation.

It’s a common issue that MySQL selects an inappropriate connection collation, regardless of the headers in the HTTP request (those are irrelevant, since it is the PHP script that connects to the database). You can either force the connection to UTF8 in the MySQL configuration, or issue two queries on every queries that set the connection to UTF.

Those would be:
SET CHARACTER SET UTF8;
SET NAMES UTF8;

You can read on those on the MySQL docs -
http://dev.mysql.com/doc/refman/5.0/en/charset-connection.html

By: Johan Sand

Johan Sand — Thu, 07 Jan 2010 22:34:19 +0000

and for Friday Fun - if you want to update the entire database including all potentially affected records in all relevant fields in all tables, then this would be a crazy kenobi option. This time only amend db host, user, pass and name. Again, upload to site and run through firefox as is. ps. make sure there's enough execution time for php to wrap it up.





	
	



query ("SHOW TABLES")) {
    while ($tables = $res_tables->fetch_array(MYSQLI_NUM) ) {

        if ($res_fields = $DB->query ("SHOW COLUMNS FROM ".$tables[0])) {

            if ($res_key = $DB->query ("SHOW COLUMNS FROM ".$tables[0]." WHERE `Key` LIKE 'PRI'")) {
                $key = $res_key->fetch_assoc();
                $unique_key = $key['Field'];
            }

            while ($fields = $res_fields->fetch_array(MYSQLI_ASSOC) ) {
                if (in_array($fields['Type'], $field_types)) {

                    $DB->query("SET NAMES latin1");

                    if ($res = $DB->query ("SELECT ".$unique_key.", ".$fields['Field']." FROM ".$tables[0]." WHERE 1=1")) {
                        while ($data = $res->fetch_object() ) {

                            $DB->query("SET NAMES utf8;");

                            $unique_field = $data->$unique_key;
                            $fix_field = bin2hex($data->$fields['Field']);

                            $result = $DB->query ("
                                UPDATE ".$tables[0]."
                                SET ".$fields['Field']." = UNHEX('".$DB->real_escape_string($fix_field)."')
                                WHERE ".$unique_key." = '".$unique_field."'
                            ");

                            unset($unique_field);
                            unset($fix_field);

                        }
                    }

                }
            }

        }
        unset($key);
        unset($unique_key);

    }
}

?>

By: Johan Sand

Johan Sand — Thu, 07 Jan 2010 08:11:30 +0000

To make a very long story short - this is what you need to do: - Amend DB specific entries (host, user, pass, db, fields and table). - Don't fiddle with the rest of the code. - Upload to browsable part of your web server/site. - Call the "page" from firefox. - Wait (and don't reload) until complete. If you need any of the code explained, feel free to drop me an email. hth, cheers. /j. ps. the code tag strips brackets, which is a bit annoying (converted to lt|gt)...





	
	



query("SET NAMES latin1");

if ($res = $DB->query ("SELECT unique_field, fix_field_1, fix_field_2, fix_field_3, fix_field_4 FROM fix_table WHERE 1=1")) {

  echo 'rows: '.$res->num_rows;
  $cnt = 0;

  while ($data = $res->fetch_object() ) {

    $DB->query("SET NAMES utf8;");

    $unique_field = $data->unique_field;
    $fix_field_1 = bin2hex($data->fix_field_1);
    $fix_field_2 = bin2hex($data->fix_field_2);
    $fix_field_3 = bin2hex($data->fix_field_3);
    $fix_field_4 = bin2hex($data->fix_field_4);

    $result = $DB->query ("
        UPDATE fix_table
        SET
            fix_field_1 = UNHEX('".$DB->real_escape_string($fix_field_1)."'),
            fix_field_2 = UNHEX('".$DB->real_escape_string($fix_field_2)."'),
            fix_field_3 = UNHEX('".$DB->real_escape_string($fix_field_3)."'),
            fix_field_4 = UNHEX('".$DB->real_escape_string($fix_field_4)."')
            WHERE unique_field = '".$unique_field."'");

    echo $cnt." - ".$unique_field."
";

    unset($unique_field);
    unset($fix_field_1);
    unset($fix_field_2);
    unset($fix_field_3);
    unset($fix_field_4);

    echo $DB->error;

    $cnt++;
    }
}

?>

By: Josue Rodriguez

Josue Rodriguez — Wed, 06 Jan 2010 23:29:07 +0000

This powerful but simple perl script is marvelous to convert your MySQL database charsets to UTF8 quick and easy. I use it every time. http://www.pablowe.net/convert_charset

By: Andreas Lagerkvist

Andreas Lagerkvist — Sat, 02 Jan 2010 11:31:18 +0000

I’m not sure if this helps, and I know some people already pointed some of it out, but I recently converted my DB to UTF-8 and this is what I did:

1. mysqldump the whole thing to a file
2. Add a special character (like “Ö”) to said file that looks good in the editor
3. Open the file with Firefox and check which encoding is used when the “Ö” looks ok (to find out exactly what encoding the file is)
4. Run iconv on the file to actually convert it to UTF-8 (from whatever encoding Firefox said it was)
5. Manually convert bad characters to good ones (and change potential encoding=latin1-settings in the sql-file to utf8)
6. Create new database where everything is UTF-8
7. Import the new, clean, utf8 SQL

That worked for me at least and I’ve had problems with encodings as far as I can remember.

I think one important bit I didn’t see in the comments (although it may have been mentioned) is to not only convert the characters but also convert the actual file (which I used iconv for).

By: Emil Björklund

Emil Björklund — Sat, 12 Dec 2009 10:01:09 +0000

After reading this article + comment thread, I’ve decided that the easiest solution to these pesky characted encoding problems is if I just change my name.

I was thinking maybe Emil Borkedchar?

By: Mike D.

Mike D. — Mon, 07 Dec 2009 02:15:30 +0000

“This whole post (and the comments) in my opinion demonstrates quite well why you should not trust your data to a database.”

Funny.

Seriously though, you’re probably already going to do this but please post a follow-up post with an overview of the problem and the eventual solution, when you find it. Going through all of these comments makes me feel like a total N00000000B. This has happened to me in WordPress a couple of times and each time I’ve just done manual search-and-replace for the characters I know about.

By: Aeron Glemann

Aeron Glemann — Fri, 04 Dec 2009 13:32:09 +0000

I’ve had to deal with this a bunch of times…. what I do – and it’s always worked for me – is 1st do a dump. Then – assuming you’re on Mac or Linux – run from the commandline:

iconv -f latin1 -t utf8 myDump.sql > myDumpUTF8.sql

Reimport….

By: Matt Sharkey

Matt Sharkey — Wed, 02 Dec 2009 23:31:17 +0000

Finally solved this problem for myself, using the method described in this post:

http://tlug.dnho.net/?q=node/276

Yes, it’s another MySQL dump & import procedure. Haven’t checked for truncated content, but so far all my em & en dashes look good.

By: Wade Kwon

Wade Kwon — Sat, 28 Nov 2009 23:55:55 +0000

Eric: Just quickly commenting to say that I’m having the exact same problem of late, and will read through the comments and any updates from you on a workable solution. Tired of doing find/replace.

By: Ash Searle

Ash Searle — Tue, 24 Nov 2009 17:08:46 +0000

@Eric,

I had to do this last year and blogged about it at the time. I remember an early draft including instructions “open vim and…” – I quickly realised as soon as you get an editor involved you’re fooked. Fortunately, MySQL has a command-line tool for doing search-and-replace so you don’t have to worry about editor settings or other random phenomena. (the instructions are in my latin1 to utf8 conversion post)

BTW. Using Safari 4.0.4 (the latest) on OS X, the encoding in this article looks fine, but the comments are screwed up. Forcing the text-encoding to ISO Latin 1 fixes the comments, but borks the names of the commenters (e.g. Tantek Çelik) I don’t know how far you think you’ve got fixing the issues, but it looks like there’s some way to go… (Note: using the web inspector / firebug you can check document.characterSet for the displayed character-set – which is handy when you’re checking you’ve overridden the text-encoding via browser menus.)

By: Jeroen Pulles

Jeroen Pulles — Sun, 22 Nov 2009 16:49:07 +0000

I had the same or similar problem last year, with a client, where my Wordpress data got encoded to UTF-8 twice. I rolled my own script to "double decode" the binary mess in my SQL dump file back to some sane text with the script that is linked above. Perhaps that can be of any help, if you're the scripting kind of person.

By: Kim Sullivan

Kim Sullivan — Sun, 22 Nov 2009 15:10:36 +0000

comments: tl;dr, just that I’ve had exactly zero succes with dumping the database (via phpMyAdmin) and reimporting.

I’ve ran across this many times, the problem is that MySQL is encoding aware, and once you get bogus data in the database, no amount of recoding between different encodings or setting “set names” will help (in fact, the worst thing you can do is try to repair it by simply setting the correct encoding for your tables – if the target encoding doesn’t have a glyph, it gets irreversibly replaced by ?).

There’s a simple workaround that worked for me many times.

First, change the database column type from TEXT (or varchar) to BLOB (or VARBINARY). This makes MySQL “forget” about any encoding it thinks the data is in, and prevents any recoding of the data that goes around behind the scenes (in my case, the data was often encoded in CP1250 or UTF-8, but column encoding was set to LATIN1).

Then you have to find out in what encoding the data is in, and reset the column type to text/varchar/char with the encoding that matches the physical encoding of the data (in my case often CP1250). Once the physical encoding in the database matches the “logical” encoding of the columns, it’s possible to simply change the encoding of the columns (to UTF-8), and with the correct SET NAMES, you can have your webpage output anything you want (from UTF-8 to LATIN1).

When the data is double encoded (it originally was in UTF-8, it got reimported into the database as latin1 and then the encoding of the columns changed to UTF-8), you first have to set the encoding of the columns back to what it was when it was imported – this changes doubly encoded UTF-8 to physically singly encoded UTF-8 that the database thinks is in LATIN1 (for example), and then you go the route from TEXT (latin1) -> BLOB -> TEXT(UTF-8).

I think I have seen some scripts that try to do this automatically (by being really smart and getting information from the data dictionary), but for smaller scale databases such as wordpress, doing everything manually might be more tedious, but I think it’s safer.

A few short points:
1. It is vital to get the physical encoding to match the encoding that is set in the table column type (I’m not sure if search and replace will help because it works on already encoded data)
2. The encoding that is set in the HTML pages only determines what encoding the browser sends to PHP
3. PHP doesn’t know (or, unfortunately, care) what encoding you get from the browser. GIGO.
4. The MySQL cares about the encoding of the data from the browser (and what encoding it sends back). Use “SET ENCODING” SQL command to tell the database this information (AFAIK, WP does this).
5. The database performs a lot of conversion behind the scenes – if the database thinks you send it data in latin1, but you have tables columns in UTF-8, it WILL do a conversion from latin1 to utf-8, even if the data already was in UTF-8 (or worse, cp1250).
6. Changing the encoding of a column from one encoding to another performs physical recoding of the data, so you have to roundtrip it via BLOB or BINARY.
7. Once you try to convert two incompatible encodings, MySQL will insert a question mark (physically) for every character it can’t convert (happens for example when changing between CP1250 and LATIN1, or importing UTF-8 data as UTF-8 data in table columns that have their encoding set to LATIN1).

By: Eric Meyer

Eric Meyer — Sat, 21 Nov 2009 02:44:44 +0000

Okay, now that's five recommendations to dump and re-import after I already tried that and it didn't work. Made things much, much worse, in fact. Tantek, I sort of agree with you, but there are things WP does for me that hand-rolling wouldn't provide. Like comments, for example, which I am emphatically not willing to outsource to a third-party cloud service; and which simply listing inbound links does not come close to replicating. Perhaps there are solutions now that would do all this but not rely on a database, but I don't remember seeing any back in 2004. Jeff, I believe that if I freshly installed WP in 2009, it would set things up using UTF-8 and there'd be no issue. I installed it almost six years ago, though. Things have advanced a bit since then.

By: Philip Tellis

Philip Tellis — Sat, 21 Nov 2009 00:21:35 +0000

This looks like a classic double-encoding problem.

ie, your data stored in your database is UTF-8, but something in between the database and your HTML output thinks that it’s LATIN-1, so goes ahead and converts it to UTF-8. The result is that multi-byte characters get treated as multiple single byte characters, each of which is converted into a single multi-byte UTF-8 character.

Skimming through your comments, it looks like your database is set to latin1. Check if your tables are also set to latin1 or utf-8 (SHOW CREATE TABLE).

If they are latin1, then what you have to do is dump all your data out as latin1 or binary (so that it doesn’t do any conversion). You could just use mysqldump to do this and it will use the default character set.

After that, you need to delete all data in the database. This can be done with by issuing a drop database. This will break your website, and probably give the first visitor the option of creating a new blog, so you might want to turn off access to your blog while this is happening.

After you’ve dropped the database, do two things. In the /etc/my.cnf file, make sure the default character sets are set to utf8, you’ll need to check the mysql website for the specific names they use for collation. I think it’s utf8_general_ci or utf8_unicode_ci. Secondly, go through the dump file in a text editor and make sure none of the tables have the latin1 charset. If they do, you need to change this as well (make a backup of the file first in case a typo gets in there).

Once done, recreate the DB using mysql < dump-file.sql

After this, all your data should still be in utf8, but now mysql actually knows it's in utf8. You can verify by doing a SHOW CREATE TABLE on all your tables and make sure they're set to utf8.

Now re-enable your blog and hope it works.

Note that Wordpress may have a way to dump the data and re-import it, but the dumped data may have the same problems that you're seeing. One option is to maybe tell wordpress that you expect latin1 data in the db and that you want your output to also be latin1. That way WP will ask MySQL for latin1 data (no conversion done) and will output latin1 data (no conversion done), and what you get is what is in the db. Then before re-importing, change the charsets everywhere to utf8.

Note that I don't use WP, so have not tested this on my own with WP.

@Tantek: while I agree with you that users who don't want to be DBAs shouldn't be writing their data to a DB that they control, the problem here is neither the database's fault, nor the user's fault. It appears to be a piece of software (WP?) making assumptions about the data and the database's character set. One of those it controls, and the other is easy to check.