Trying to match word repetition, an issue with match overlap?

[bew]

Contributing guidelines

• I have read CODE_OF_CONDUCT.md

Module(s)

mini.hipatterns

Question

Hello!
First of all, I absolutely love this plugin! ✨

Second, I want to highlight any number of repetition of TODO that are separated by underscores.
For example TODO_TODO_TODO in TODO_TODO_TODO_something.

I tried using patterns like this:

    {
      -- e.g. TODO_TODO_TODO_something
      "()TODO_()TODO_", -- a _TODO_ followed by (at least) another _TODO_
      "TODO_()TODO()_[^T][^O]", -- the final _TODO_ of the chain (NOT followed by another _TODO_)
    },

But I'm only getting this:

So the first pattern matches, the second also matches, but I would expect the first pattern to re-match for the middle TODO_ as well 🤔

[drowning-cat]

1. What a crazy example, hard to figure out what you actually want.
2. Just use "TODO_" for simplicity.
3. Or, using your logic, you could try { "()TODO_()TODO_", "TODO_()TODO_()" }.

[bew]

1. Oh sorry yes, I updated my post with:

   I want to highlight any number of repetition of TODO that are separated by underscores.

2. The issue with that is that it's going to highlight the last _ before something which I don't want 👀

3. Well this gives another wrong highlight..
   

Inspired by (3.) I also tried "()_?TODO()" and that works to highlight my example,
BUT it also highlights an italic _TODO_ in markdown which I don't want.
(and it would also highlight FOO_TODO which I don't want) 😬
This above issue is why I tried to match a chain of TODOs instead

[drowning-cat]

Note

I don’t think pattern matching can do this without drawbacks.

I suppose this is what you want:

1. Easy

MiniHipatterns.setup({
  highlighters = {
    todo = { pattern = { --[["TODO",]] "TODO_TODO", "TODO_TODO_TODO", --[[...]] }, group = "MiniHipatternsHack" },
  },
})

2. Hard

MiniHipatterns.setup({
  highlighters = {
    todo = {
      pattern = { "TODO_", "_TODO" },
      group = function(b, m, d)
        local line = vim.api.nvim_buf_get_lines(b, d.line - 1, d.line, true)[1]
        local range = { d.from_col, d.to_col }
        if m == "_TODO" then
          local left = line:sub(range[1] - #m + 1, range[1])
          if not left:match("TODO_") then return nil end
        end
        if m == "TODO_" then
          local right = line:sub(range[2], range[2] + #m - 1)
          if not right:match("_TODO") then return nil end
        end
        return "MiniHipatternsHack"
      end,
    },
  },
})

[echasnovski]

So the first pattern matches, the second also matches, but I would expect the first pattern to re-match for the middle TODO_ as well 🤔

The core reason for the expectation mismatch is indeed how several pattern matches on a single line is done. The current approach works like this:

• Find full match of the whole pattern (not only intended submatch). Register pattern's (sub)match.
• Keep looking for the next pattern match starting from the right of the full match.

This is how Neovim's search (like with /) works. With a string of many -, searching for -- will not result into overlapping matches.

What would make this use case work with a single TODO_TODO pattern is to keep looking starting from the right of the whole initial match start.

In theory, this is a very trivial change of to + 1 into from + 1. But I think this was not done initially to 1) "be more like regular search matches"; and 2) performance reasons (each line part is searched only once). There might have been some edge cases also, but they are at least not in the tests.

I'll take a closer look at what is the best compromise here. An alternative approach is to keep looking fro the right of the real match. At least it makes it work with the patterns in the original comment.

[echasnovski]

Oh that's not at all what I had envisioned, I'll try to take a look at the code and propose a PR if I can!

No need. I am taking a look myself.

[bew]

The basis of my idea is to pre-process the pattern to see if it's a plain xxx or a pattern with submatches

Then for the hot loop:

• for plain patterns: use current matching and advancement
• for patterns with lookbehind, advance from +1
• for other patterns, advance sub_to +1

[echasnovski]

Although might be reasonable, this looks too complicated. At least to document. I'd rather choose a single approach and suggest to use callable group to adjust anything incompatible with it.

[bew]

Soo, I'm not sure about your position exactly: are you going to make a change to make this possible in a sensible way using group ?
Or would you suggest the example above as how you'd recommend implementing this?
(I don't particularly like his impl, it's not very explicit and looks very much like a hack.. 👀)

If not, I'm tempted to fork hipatterns to make my own version, with more flexible side-effects & matching selection 🤔

[echasnovski]

Soo, I'm not sure about your position exactly: are you going to make a change to make this possible in a sensible way using group ?

My proposition is to let me think and explore more about it. Currently I am inclined to implement sub_to + 1 version, i.e. "keep looking from the right of the real match", which will make this use case possible with patterns from the initial comment.

[echasnovski]

This should now be resolved on latest main. The patterns from the original comment should work now, as described in this note.

[drowning-cat]

require("mini.hipatterns").setup({
  highlighters = {
    repeat_todo = {
      pattern = { "()TODO()", "TODO()_()TODO" },
      group = "MiniHipatternsTodo",
    },
  },
})

We can also use:

{ "()TODO()_", "_()TODO()", "TODO()_()TODO" }

[bew]

Awesome, thank you! 🙏