← Reference · Nestor G Pestelos Jr · Print this page

Programming Languages · Ruby

Ruby String Tokenization

Reference entry · last updated September 21, 2026

Ruby string tokenization turns a body of text into a list of normalized words. The usual pipeline splits the text on whitespace with String#split, lowercases each piece, removes leading and trailing characters that are not letters, and discards anything that becomes empty. Each step is a pure string operation that returns a new string, so the original text is unchanged.[1][2]

First principles and definitions

The split algorithm

String#split divides a string into substrings at a delimiter. The delimiter selects one of two algorithms. An explicit pattern marks a boundary at each match. When the pattern is nil, which is the default, the value of $; is used, and because that global is nil by default the string is split on whitespace as if a single space had been given. Splitting on whitespace ignores leading and trailing whitespace and treats a run of contiguous whitespace as one separator, so no empty strings appear at the edges.[1]

Two consequences follow. An empty input returns an empty Array, because the string has no fields to split. And a token that is entirely punctuation becomes an empty string after normalization, so it must be dropped rather than counted.

"  The  law ".split   # => ["The", "law"]
"".split              # => []
"!!! ... !!!".split   # => ["!!!", "...", "!!!"]

Anchors and the edge of a string

A pattern that removes edge characters must say which edge it means. \A matches the start of the string and \z matches the end, while ^ and $ match the start and end of a line. Tokens produced by whitespace splitting contain no newlines, so the difference does not show there, but using the string anchors states the intent exactly.[3]

Because String#gsub replaces every match, a single pattern with two alternatives can remove a leading run and a trailing run in one pass. The alternatives are a run of non-letters at the start, or a run of non-letters at the end.

Splitting on whitespace

Calling split with no argument is equivalent to split(' ') for ordinary input and is the normal way to get words. Both forms collapse runs of whitespace and strip the edges.[1]

"The law governs contracts.".split
# => ["The", "law", "governs", "contracts."]

Note that "contracts." keeps its period. Splitting decides where tokens end, but it does not remove punctuation attached to a token. That is the job of the normalization step.

Normalizing tokens

String#downcase folds case so that "The" and "the" count as one word. String#strip removes leading and trailing whitespace and takes no argument; it removes only whitespace, so it does not touch punctuation and it cannot be given a pattern. Removing punctuation from the edges requires a substitution.[2]

"  The  ".downcase.strip        # => "the"
"(governs)".downcase.gsub(/\A[^a-z]+|[^a-z]+\z/, '')
# => "governs"

The pattern reads as two alternatives. \A[^a-z]+ matches a run of characters that are not lowercase letters at the start, and [^a-z]+\z matches such a run at the end. gsub removes both. Punctuation inside a token is left alone, which is why an internal apostrophe or hyphen survives.

A token that was only punctuation becomes an empty string, so the loop that consumes the tokens must skip it.

Character classes

The class \W means a character that is not a word character, and a word character includes letters, digits, and the underscore. That is wider than the rule "not a letter", so a token such as a bare number would not be stripped by \W. The class [^a-z] states the narrower rule directly, and it is correct after downcase has removed the upper case letters.[3]

The examples here use the ASCII class [^a-z]. Text in other scripts needs [[:alpha:]] or an equivalent Unicode class, and case folding for those scripts is a separate question.

A full pipeline

Combining the steps gives a counting routine over normalized tokens. The split produces raw tokens, each is normalized in place, empty results are skipped, and a hash accumulates the counts.

counts = Hash.new(0)

"The law governs contracts. The parties govern the parties!".split.each do |raw|
  token = raw.downcase.gsub(/\A[^a-z]+|[^a-z]+\z/, '')
  next if token.empty?
  counts[token] += 1
end

counts
# => {"the"=>3, "law"=>1, "governs"=>1, "contracts"=>1, "parties"=>2, "govern"=>1}

Ordering the result is a separate concern; see Ruby Composite Sort Keys for count descending with an alphabetical tiebreak, and Ruby Hash Accumulators for the accumulator choice.

Choosing the rules

• Words separated by any whitespace: text.split.
• Case-insensitive comparison: downcase before counting.
• Remove edge punctuation only: gsub(/\A[^a-z]+|[^a-z]+\z/, '').
• Remove all punctuation inside a token too: use a single class, gsub(/[^a-z]/, ''), which joins fragments: mother-in-law becomes motherinlaw.
• Non ASCII text: use a Unicode letter class instead of [^a-z].

References

1. ^ Ruby 2.6.10 core documentation, "String," split. Free full text: ruby-doc.org/core-2.6.10/String.html
2. ^ Ruby 2.6.10 core documentation, "String," downcase, strip, and gsub. Free full text: ruby-doc.org/core-2.6.10/String.html
3. ^ Ruby 2.6.10 core documentation, "Regexp," anchors and character classes. Free full text: ruby-doc.org/core-2.6.10/Regexp.html