Skip to content

Prevent crawler from changing url #85

@Pau1fitz

Description

@Pau1fitz

Do you want to request a feature or report a bug?

What is the current behavior?

When using the crawler I have a url with an ! included the following url gets created:
https://www.test.com/liverpool/players%21/

however, I am looking for the url to look like the following:

https://www.test.com/liverpool/players!/

If the current behavior is a bug, please provide the steps to reproduce.

My code looks like the following:

const SitemapGenerator = require("sitemap-generator");

// create generator
const generator = SitemapGenerator("http://www.test.com", {
    stripQuerystring: false,
    decodeResponses: true
});

// register event listeners
generator.on("done", () => {
    // sitemaps created
    console.log("sitemaps created");
});

// start the crawler
generator.start();

When running node generator.js the url is being replaced with %21 for !

What is the expected behavior?

I want to keep the original url i.e. - https://www.test.com/liverpool/players!/

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions