How to remove duplicates from resource

  • Hey

    Im trying to create a parser very similar to his one:

    In this tutorial it is showed how to create parser.. but it has the same problem as i have.. it parses the same url 2-3 times and it adds it to resource.. so it means it will visit the same url 3 times.. which is useless for me..

    What i want to do is to scrape url (even if it exist several times on page) but adds to resource it only ones.. or it has to automatically deletes any duplcaite before visiting specific page

    Thx in advance

  • I have another problem with this parser

    1. i have created resource where all urls will be added (like in tutorial)
    2. In logs it seems that the script is working .. its showing that it is scraping all urls on page
    3. In resource i have only 1 url.. the first one.. why it added only the first url and not the rest?

Log in to reply