How to find all matches in a string with re.findall in Python

The re.findall function is a simpler yet powerful tool for extracting all occurrences of a pattern from a string. At its core, it scans through the target text and returns a list of matches, making it perfect for when you want to pull out every piece of data that fits a certain criteria.

Its simplest usage involves passing a pattern and a string to search, like this:

import re

text = "apple banana cherry apple"
matches = re.findall(r"apple", text)
print(matches)
# Output: ['apple', 'apple']

Notice that re.findall returns a list of strings that matched the pattern, not a list of match objects. That’s useful for quick extraction, but it also means if you want more detailed information like the position of the match, you’ll need a different approach, such as re.finditer.

Patterns can be simple literals or more complex regular expressions. For example, to find all words starting with a vowel:

text = "apple banana orange grape"
vowel_words = re.findall(r"b[aeiouAEIOU]w*", text)
print(vowel_words)
# Output: ['apple', 'orange']

The key here is the use of anchors like b for word boundaries and character classes like [aeiouAEIOU]. This allows re.findall to zero in on precisely the segments of text you care about.

One subtlety to watch out for is when your pattern contains groups. In such cases, re.findall returns a list of tuples corresponding to the groups, instead of the full matched string. For example:

text = "Name: Alice, Age: 30; Name: Bob, Age: 25"
pattern = r"Name: (w+), Age: (d+)"
results = re.findall(pattern, text)
print(results)
# Output: [('Alice', '30'), ('Bob', '25')]

Each tuple contains the captured groups, which can be very useful for structured data extraction but requires a shift in how you process the results.

When you don’t use groups, the entire matched text is returned. With groups, you get just the parts inside the parentheses. This means your pattern design directly impacts the shape of the data you receive.

Remember that re.findall always returns substrings that matched the pattern, never the unmatched parts of the string. If you want to split on a pattern rather than find matches, re.split is the appropriate function.

Also, by default, re.findall works in a case-sensitive manner unless you explicitly specify the re.IGNORECASE flag:

text = "Hello hello HeLLo"
matches = re.findall(r"hello", text, re.IGNORECASE)
print(matches)
# Output: ['Hello', 'hello', 'HeLLo']

This flag is essential when the case does not matter for your extraction logic.

In summary, re.findall is the go-to method when you want to extract all occurrences of a pattern as strings or tuples of groups. Its simplicity is its strength, but it requires careful pattern crafting to get the exact form of data you need. Next, understanding how groups influence output will help you tackle more complex patterns effectively.

Handling complex patterns with groups

Groups in regular expressions provide a mechanism to isolate specific parts of a match, which will allow you to extract just the components you care about. However, when you use multiple groups, the result from re.findall becomes a list of tuples, with each tuple containing the contents of the captured groups in order.

Consider a log file where each entry captures a timestamp, a log level, and a message:

log_data = """
2024-04-01 12:00:00 INFO User login successful
2024-04-01 12:05:23 ERROR Unable to connect to database
2024-04-01 12:07:45 WARNING Disk space low
"""

pattern = r"(d{4}-d{2}-d{2} d{2}:d{2}:d{2}) (w+) (.+)"
matches = re.findall(pattern, log_data)
for timestamp, level, message in matches:
    print(f"At {timestamp}, a {level} occurred: {message}")

This outputs:

At 2024-04-01 12:00:00, a INFO occurred: User login successful
At 2024-04-01 12:05:23, a ERROR occurred: Unable to connect to database
At 2024-04-01 12:07:45, a WARNING occurred: Disk space low

Here, each tuple neatly separates the components of the log entry, making it simpler to work with structured data extracted from free-form text.

It’s also possible to mix capturing and non-capturing groups. Non-capturing groups, written as (?:...), allow you to group parts of the pattern without extracting them:

text = "foo123bar foo456baz"
pattern = r"foo(?:d+)(bar|baz)"
matches = re.findall(pattern, text)
print(matches)
# Output: ['bar', 'baz']

In this example, the digits are grouped for pattern matching but not captured. Only the suffix bar or baz is returned. This keeps the output focused on what’s relevant.

Another nuance arises with nested groups. When groups are nested, re.findall still returns tuples containing all groups, including inner ones. For example:

text = "abc123def456"
pattern = r"(abc(d+))"
matches = re.findall(pattern, text)
print(matches)
# Output: [('abc123', '123')]

The outer group captures abc123 and the inner group captures just the digits 123. This can lead to more complex tuples, requiring you to carefully unpack or process results.

When you want to extract overlapping or adjacent matches with groups, keep in mind that re.findall always finds non-overlapping matches from left to right. For overlapping matches, you’ll need to use lookahead assertions or switch to re.finditer for finer control.

Lookahead can be employed to capture overlapping patterns without consuming characters:

text = "ababab"
pattern = r"(?=(aba))"
matches = re.findall(pattern, text)
print(matches)
# Output: ['aba', 'aba']

This pattern uses a positive lookahead to find all occurrences of aba including overlapping ones, by asserting the pattern without advancing the match position.

Groups also enable conditional logic within regexes, but that’s a more advanced topic. For now, understanding how groups affect the shape and content of re.findall’s output is key to using it effectively.

Optimizing performance for large text searches

When dealing with large volumes of text, performance becomes a critical consideration. The re.findall function, while convenient, can become a bottleneck if the regex pattern or input size is not optimized. To enhance performance, it’s important to understand how the regex engine processes patterns and how to write efficient expressions.

One common performance pitfall is using overly broad or ambiguous patterns that cause excessive backtracking. For example, a pattern like .* or nested quantifiers can dramatically slow down matching on large strings:

import re

text = "a" * 10000 + "b"

# Inefficient pattern with greedy quantifier
pattern = r"a.*b"
match = re.findall(pattern, text)
print(match)

In this case, the engine tries to match as much as possible with .* before backtracking to find the trailing b. This can be very slow on large inputs.

A better approach is to use non-greedy quantifiers or more precise character classes to limit backtracking:

pattern = r"a.*?b"
match = re.findall(pattern, text)
print(match)

The .*? makes the match non-greedy, so the engine stops at the first possible b, reducing unnecessary backtracking.

Another strategy for optimizing searches is to compile your regex pattern once if you’re applying it multiple times. Compiling avoids recompiling the pattern on each call, which can save time in loops or repeated operations:

import re

text = "some large text ... " * 1000
pattern = re.compile(r"bw{4,6}b")  # words of length 4 to 6

matches = []
for _ in range(100):
    matches.extend(pattern.findall(text))

That is significantly faster than calling re.findall with a string pattern repeatedly, especially in performance-critical code.

Using flags such as re.ASCII instead of the default Unicode matching can also speed up matching when you know your input is ASCII-only. This reduces overhead in character class evaluations:

pattern = re.compile(r"w+", re.ASCII)
matches = pattern.findall(text)

This can make a difference when processing very large data sets.

When extracting multiple different patterns from a large text, consider combining them into a single regex with alternations rather than running multiple separate findall calls. This reduces the number of scans over the text:

text = "error 404, warning 300, info 200"
pattern = re.compile(r"error d+|warning d+|info d+")
matches = pattern.findall(text)
print(matches)
# Output: ['error 404', 'warning 300', 'info 200']

This approach also makes it easier to maintain and optimize your regex logic.

Finally, for extremely large files or streams, consider processing the text incrementally rather than loading it all into memory. Using re.finditer allows you to iterate over matches lazily, which can keep memory usage manageable:

import re

with open("large_log.txt") as f:
    pattern = re.compile(r"ERROR: (.+)")
    for line in f:
        for match in pattern.finditer(line):
            print(match.group(1))

In this way, you combine efficient memory use with the power of regex extraction, which is especially important for real-world applications dealing with big data.