Document content type filter

[ribenamaplesyrup]

If we download all files from an organisation's website we are likely to encounter a mix of documents including publications, job ads, workplace policies, newsletters etc. If we wanted to only analyse one type of document (for example publications) we should add a step to filter out the documents we're not interested in. As I see it there are two approaches we could take to filter out irrelevant documents:

Filter by parent link

Only retrieve/analyse documents found on webpages that contain a specific base url.

Filter using LLM

Only retrieve/analyse documents that an LLM identifies as a research publication.

[ribenamaplesyrup]

'Filter by parent link' won't always be a useful filter as the webpage a publication PDF file is embedded within may not contain a useful base url for filtering. Worth instead prioritising LLM filtering based on document content.