Updates to Chapter 14, “Strings and sorting”

[This post notes differences between the fifth and sixth editions.]

I did quite a bit of work to update Chapter 14, but most of it isn’t going to show up in that chapter. I initially added a long section on Unicode normalization forms, covering the difference between canonical and compatibility forms. You need to know these to properly sort Unicode strings (see Know your sort orders over at The Effective Perler).

As I went through the explanation, though, I realized that I was also going to need the same concepts for the basic Perl strings, and also for the regular expressions chapters. Even basic comparisons need the idea of equivalence, and the regular expressions even more so (and, there might soon be a /k match flag that will do that for us).

While I was writing this chapter, which includes a section on index(). Since Perl’s string operators work on characters instead of grapheme, could I find accents that way. This material didn’t make it into the book.

use utf8;

my $string = "éáabcáá\x{65}\x{301}í";    

my( $pos, $old_pos );
while( -1 != (my $pos = index $string, "\x{301}", $old_pos + 1 ) ) {
	print "Found accent at $pos\n";
	$old_pos = $pos;
	}

Since I’ve specified only one decomposed é, I get only one match:

Found accent at 8

It doesn’t find the other accents though. I could decompose the string:

use utf8;

use Unicode::Normalize;

my $string = "éáabcáá\x{65}\x{301}í";    

my $decomposed = NFD( $string );

my( $pos, $old_pos );
while( -1 != (my $pos = index $decomposed, "\x{301}", $old_pos + 1 ) ) {
	print "Found accent at $pos\n";
	$old_pos = $pos;
	}

Now I can tell that there are accents, although the positions no longer have much meaning because they don’t relate to the original string:

Found accent at 1
Found accent at 3
Found accent at 8
Found accent at 10
Found accent at 12
Found accent at 14

To get around that, I have to do a lot more work. I can go through each grapheme individually, decompose each one, and look at that:

use 5.012;
use utf8;
binmode STDOUT, ':utf8';

use Unicode::Normalize;

my $string = "éáabcáá\x{65}\x{301}í";    

my @graphemes = $string =~ m/(\X)/g;

while( my( $index, $grapheme ) = each @graphemes ) {
	my $decomposed = NFD( $grapheme );
	print "Found an accent at $index ($grapheme)\n"
		if -1 < index( $decomposed, "\x{301}" );
	}

That's fine, and it reports the right positions for the characters that have accents:

Found an accent at 0 (é)
Found an accent at 1 (á)
Found an accent at 5 (á)
Found an accent at 6 (á)
Found an accent at 7 (é)
Found an accent at 8 (í)

Notice that I use the new form of each on arrays in Perl 5.12.

That's not a very good way to do it though because I'm only looking for the ´mark. I should look for any mark:

use strict;
use warnings;

use utf8;
binmode STDOUT, ':utf8';

use Unicode::Normalize;
	
my $string = "éáåbcüá\x{65}\x{301}í";

my $array = [$string =~ m/(\X)/g];
while( my( $index, $grapheme ) = each $array ) {
	my $nfd = NFD( $grapheme );
	print "Found an accent at $index ($grapheme)\n"
		if $nfd =~ /\p{Mark}/;
	}

Now I can find all sorts of marks:

Found an accent at 0 (é)
Found an accent at 1 (á)
Found an accent at 2 (å)
Found an accent at 5 (ü)
Found an accent at 6 (á)
Found an accent at 7 (é)
Found an accent at 8 (í)

There are easier ways to do this, but I wanted to stick to just what was in Learning Perl.