Description:

The extraction does however stop after 3 pages and does not cycle through all the pages. Is this a restriction because I am using the free version?

A:

You can scrape a website with unlimited Page/URL inquiries with Octoparse free version.

Every time you encountered this situation, you may need to check the XPath for the pagination loop from Octoparse.

In most cases, the original XPath won't be able to correctly locate to the Next Page link at the bottom of the web page.

For example, the screenshot below shows the situation that the original XPath cannot match the Next page element when on the 3rd page.

In this case, you need to generate the right XPath expression to find the elements (Next Page) for the pagination loop.

FireBug and FirePath are good extensions for beginner to learn XPath.

Check out the materials to learn XPath.

You can enter the correct XPath into the text box of the Single element(See the screenshot below). 

1. Select the Loop Item/Cycle Pages box in the workflow

2. Select Single element for the Loop mode 

3. Enter a correct XPath to find the Next Page element

4. Click Save 

5. Check the workflow and see if it can keep clicking the Next Page button

No Next Page link at the bottom of the web page?

If you need to click the page number to scrape multiple web pages, it's fine to use "following-sibling::" to write the XPath for finding the page next to the current page. Just enter the correct XPath into the text box of the Single element.

If it's still hard for you to use XPath, drop us a message and ask our support team for help.

We're happy to help!