[Feature]: Enhance Parsers

[init27]

Is your feature request related to a problem? Please describe.

We have scope to improve the parsers and have better ingestion at source:

• HTML parsing
• PDF parsing
• DOCX parsing
• PPT Parsing

Describe the solution you'd like

Better solutions for ingest command

Describe alternatives you've considered

No response

Additional context

No response

[Lucas-Fernandes-Martins]

Hi @init27 , I spent some time investigating and there seems to be a mismatch between the PPT and DOCX parsers: they should return a List object in the format (to be compatible with .lance):

return [{"text": text}]

But instead they return a string (see here):

return "\n\n".join(all_text)

PDF parser returns a list and is already correct.

I already fixed both PPT and DOCX to return the right format and it seems to be working now.

If you're ok with it I can open a PR with this small fix.

Regarding a parser for HTML, what's your vision for it? Would it make sense to treat it in the same way text files are treated?

Thank you :)

[vichustephen]

Hi, @Lucas-Fernandes-Martins There is same issue for TXT parser also incompitable with lance dataset

[Lucas-Fernandes-Martins]

Thank you @vichustephen just updated my code to reflect that. I hope to submit a PR soon.

[init27]

@Lucas-Fernandes-Martins thanks for working through this and sending a PR!

I am also thinking if there are better parsers for PDF ingestion-right now the parsing in for PDF and HTML is pretty basic. Would you be interested in looking at improving the ingestion step as in the text that gets extracted?

Right now, its a lot of mixed formatting characters etc

(Apologies for the slow reply, I was on PTO)

[Lucas-Fernandes-Martins]

Thank you @init27 for merging the PR! I'm definitely interested in researching alternative ingestion options! I'll do some research and I'll get back to you :)

[init27]

Thanks again for being a central contributor and always helping the codebase!