WP Content Crawler

WP Content Crawler

Get content from almost any site to your WordPress blog, automatically!
For what it can be used

  • Create a personal site which collects news, posts, etc. from your favorite sites to see them in one place
  • Use it with WooCommerce to collect products from shopping sites
  • Collect products from affiliate programs to make money
  • Collect posts to create a test environment for your plugin/theme
  • Collect plugins, themes, apps, images from other sites to create a collection of them
  • Keep track of competitors

How it works
It’s all about CSS selectors and you can learn how to use them in minutes by watching the introduction tutorial.
See it in action, learn in minutes
Available in two languages, ready for translation Video tutorials


PHP >= 5.5, mbstring

Tested with WP versions:

4.5.3, 4.4.2, 4.3.3, 4.2.7, 4.1.10, 4.0.10, 3.9.11

Main features:

  • Save post’s title, excerpt and content.
  • Save meta keywords and description.
  • Save images in the posts to your site.
  • Save featured image of target post as the featured image of the post created in your site.
  • Save the list in posts (extract the list from the post).
  • Save paginated posts. If target post is paginated, the post in your site can be paginated as well.
  • Save anything from target page as post meta.
  • Find and replace anything in the target page’s HTML before saving its content.
  • Find and replace using regular expressions.
  • Find and replace in image URLs before saving them.
  • Manipulate target page’s HTML before saving to make it suitable for your needs.
  • Remove unnecessary elements from target page’s HTML.
  • Map categories of your site to target site’s categories to keep posts organized.
  • Automatically get category URLs for category mapping.
  • Add unlimited categories for a site and automatically check for new posts uniformly.
  • Set alternative CSS selectors for each setting. By this way, get content from the pages of the same site which are designed differently.
  • Automatically save posts.
  • Automatically check for new posts.
  • Set maximum number of category pages to be checked automatically.
  • Add unlimited sites.
  • Import/export site settings.
  • Set post type. For instance, you can add posts as products if you use WooCommerce.
  • Set a template for the post.
  • Set a template for each list item.
  • Set passwords for created posts.
  • Keep notes for each site to keep a changelog or anything you want.
  • Test any CSS selector and find-and-replace setting right away.
  • Test a site before activating it for automatic crawling.
  • Manually create posts by providing the URL of the posts.
  • If you want, use settings different than general settings for each site.
  • Activate/deactive each site any time to start/stop automatic crawling.
  • Either directly publish the posts or keep them as drafts to make changes before publishing.
  • Online documentation
  • Video tutorials for a quick start
  • Ready to translate (.po file)
  • Get updates from your WordPress admin panel.
  • Save images as gallery. If you use WooCommerce, you can save images from target page as product gallery.
  • Get anything from target page as a shortcode and use the shortcodes in templates to place any data anywhere you want.
  • Save tags using CSS selectors.
  • Find and replace in post tags.
  • Limit the number of tags that can be added to a post.


v1.1.3 - 26 June 2016
* Fixed: Fetching stops if there is a URL that cannot be fetched.
* Fixed: JS error saying that sortable is not a function.
* Fixed: Tag selector results are taken as single results. Make it multiple and get all of the results found by each selector.
* Fixed: Storage directory's CHMOD should be 755 to prevent any security breaches.
* Improvement: License key input field is now shown as password.
* Improvement: Each post will now have _wpcc_post_url post meta that stores the URL of the post.
* Improvement: If you leave any URL in category map empty, you will get an error.
* Improvement: Now you will get a notification when your PHP version is lower than 5.5.
* Improvement: Now you can safely deactivate plugin from license key settings page as well, which also removes license key from your current domain and makes it available to be used on other domains. v1.1.2 - 21 May 2016
* Fixed: Crawler stops saving posts although there are URLs in the queue.
* Fixed: Crawler stops crawling if there is no category URL in an active site. v1.1.1 - 16 May 2016
* New Feature: Save tags using CSS selectors
* New Feature: Find and replace in post tags
* New Feature: Limit the number of tags that can be added to a post
* Fixed: Export-settings field should not be send with the request.
* Fixed: 'Failed to open stream for upgrade.php' error caused by DatabaseService.php
* Fixed: Category list causes an error when WooCommerce is not installed. v1.1.0 - 15 May 2016
* New Feature: Create custom short codes for CSS selectors and place any element or attribute of any element at anywhere in the post template.
* New Feature: Remove unnecessary elements after the post template is ready.
* New Feature: Add images as a gallery. You can also add image gallery as a WooCommerce product's image gallery.
* Improvement: Regex errors are now shown at details section in Tester page.
* Improvement: Content existence is controlled in a smarter way.
* Improvement: You can now use your license on another site after you deactivate the plugin on the site which the plugin is already installed.
* Fixed: Firefox does not let copying texts from disabled textareas.
* Fixed: Quick-edit deletes all settings.
* Fixed: Select inputs in custom general settings are not assigned their value from database.
* Fixed: Export-settings textarea should be "readonly".
* Fixed: Hard-coded directories should be defined dynamically.
* Fixed: CHMOD of storage directory should be set as 777.
* Fixed: The results of post content selectors are not ordered by the positions of the found elements.
* Fixed: Regular expressions are not imported appropriately when importing settings.
* Fixed: Regex errors cause whole template to be empty.
* Fixed: Template editors remove HTML tags.
* Fixed: Category of the post is not assigned as selected when saving a URL more than once using manual crawling.
* Fixed: Database tables are not created on multisite.
* Fixed: Some sites are not available to select in tools page. v1.0.1 - 9 May 2016
* Fixed: MySQL does not let thumbnail_url be null.

More items by turgutsaricam

WP Content Crawler