Getting data from XML feeds using PHP

A how-to guide by Bobulous.

Introduction

[Note: This article has been superseded by the newer article reading XML feeds using SimpleXML.]

In a previous article, I introduced a set of simple functions that grab data from XML files. The functions are a little rough and ready, but with well-structured XML data formats they work very well. This article offers an example of how to use those functions to grab the latest headlines from a real-world RSS feed and then format them into a "latest news" box for your website.

The desired result

Here is the result of fetching an XML feed, extracting the desired data from it, and formatting that data into a block of HTML. Using CSS, you can alter the appearance of the news box radically, choosing colours, sizes, positions and fonts to suit the situation.

Articles: Digital Photography Review (dpreview.com)

Adobe Camera Raw 8.6 and DNG Converter 8.6 now available
Adobe has made final release versions of Camera Raw 8.6 and DNG Converter 8.6 available. This update adds support for the Nikon D810, Panasonic Lumix AG-GH4 (GH4 with interface unit) and Panasonic Lumix DMC-FZ1000, as well as several...
[19:16, 31st July GMT]
Canon India teases 'something big' coming soon
Canon India has posted a teaser on its Facebook page showing a 1-series DSLR alongside the words 'Get your camera ready, something BIG is coming'. Being a teaser rather than a leak, the meaning of the post is open to interpretation,...
[17:35, 31st July GMT]
Sony brings Cyber-shot DSC-WX220 and W810 to the US
The Sony Cyber-shot DSC-WX220 and W810, which have been available in certain markets for several months, are making their way onto US shores in August and September. The WX220 features an 18.2MP BSI-CMOS sensor, Bionz X processor, 25-250mm...
[17:00, 31st July GMT]
Hollywood studios strike deal to keep Kodak movie film in production
A group of big name Hollywood directors has nudged studio executives into talks with Kodak, negotiating a deal that would keep the company's movie film in production. With support from the likes of Quentin Tarantino, Christopher...
[00:30, 31st July GMT]
With greater clarity: Nikon D810 Shooting Experience
The Nikon D810 represents a consolidation of its two predecessors, the D800 and D800E. The 36MP full frame DSLR lacks an anti-aliasing filter to make the absolute most of its high resolution sensor and provides a number of meaningful updates over...
[19:32, 30th July GMT]

Feed fetched 2014-07-31 23:38:14 UTC

Step one: Get the RSS feed

Before you create a script that fetches an XML feed from someone else's site, make sure to check that you have permission to do so. Look for a usage policy in the feeds section of the site, or email the site owner to make sure they're happy with what you intend to do.

For this article, I'm using the latest news feed from Digital Photography Review, one of my favourite sites. The XML feeds at dpreview.com are in the RSS 2.0 format, but the techniques used in this article could be applied to most XML data formats.

Once you have the URL of the XML feed that you are going to use, you need to have PHP load the contents of the feed into a string variable. Using file_get_contents, you could fetch the XML file like so:

$xml = file_get_contents('http://www.dpreview.com/feeds/news.xml');

However, this requires that PHP is setup with allow_url_fopen set to true. Not all web hosts enable this setting, for security reasons. So another way to fetch the XML file into a string is by using the cURL functions (if they are installed on your PHP setup) like this:

// Use cURL to get the RSS feed into a PHP string variable.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,
        'http://www.dpreview.com/feeds/news.xml');
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$xml = curl_exec($ch);
curl_close($ch);

Either method should give you a string called $xml that contains the contents of the entire XML feed.

Step two: Extract data from the feed

Most XML data files contain a large amount of content. For instance, the dpreview.com news feed contains channel (feed) title and image information, a couple of dozen news items, each with a title, description, image for the item, link to visitor comments, and the publication date. To keep things simple, I'm only going to extract the channel title and URL, and the title, description and date of each news item.

Starting off with the channel details, we can grab what we want using the value_in function. (You need to include the functions in xml_regex.php, which you can download from my page about extracting XML data with regular expressions.)

// Include the handy XML data extraction functions.
include 'xml_regex.php';
// An RSS 2.0 feed must have a channel title, and it will
// come before the news items. So it's safe to grab the
// first title element and assume that it's the channel
// title.
$channel_title = value_in('title', $xml);
// An RSS 2.0 feed must also have a link element that
// points to the site that the feed came from.
$channel_link = value_in('link', $xml);

Next we want to build an array that contains individual news item elements. This can be done using the element_set function.

// Create an array of item elements from the XML feed.
$news_items = element_set('item', $xml);

Once we've got the array of item elements, we can iterate through the array, one item at a time, and extract just the data that we're interested in from each news item: the URL, title, description and date. Then we can store that data in a new array called $item_array.

foreach($news_items as $item) {
    $title = value_in('title', $item);
    $url = value_in('link', $item);
    $description = value_in('description', $item);
    $timestamp = strtotime(value_in('pubDate', $item));
    $item_array[] = array(
            'title' => $title,
            'url' => $url,
            'description' => $description,
            'timestamp' => $timestamp
    );
}

Now we have $item_array, which is an array that contains news items in the form of associative arrays, so that the title of the first news item should be found in $item_array[0]['title'] and the title of the second news item should be found in $item_array[1]['title'] and so on.

Step three: Markup the data with HTML

With all the data we want stored in one handy array, we can iterate through the array and markup the data with HTML.

if (sizeof($item_array) > 0) {
    // First create a div element as a container for the whole
    // thing. This makes CSS styling easier.
    $html = '<div class="rss_feed_headlines">';
    // Markup the title of the channel as a hyperlink.
    $html .= '<h2 class="channel_title">'.
            '<a href="'.make_safe($channel_link).'">'.
            make_safe($channel_title).'</a></h2><dl>';
    // Now iterate through the data array, building HTML for
    // each news item.
    $count = 0;
    foreach ($item_array as $item) {
        $html .= '<dt><a href="'.make_safe($item['url']).'">'.
                make_safe($item['title']).'</a></dt>';
        $html .= '<dd>'.make_safe($item['description']);
        if ($item['timestamp'] != false) {
		    $html .= '<br />' .
                    '<span class="news_date">['.
                    gmdate('H:i, jS F T', $item['timestamp']).
                    ']</span>';
        }
        echo '</dd>';
        // Limit the output to five news items.
        if (++$count == 5) {
            break;
        }
    }
    $html .= '</dl></div>';
    echo $html;
}

The above code creates the HTML for the first five news items, plus the channel title, and wraps it all in a div element with the class "rss_feed_headlines" so that CSS stylesheets can target the news box specifically.

Step four: Consider security implications of external data

I trust the people at dpreview.com not to send anything malicious to me via their RSS feeds. But suppose their site was hacked or infected by malware. Then they wouldn't be in control of what their RSS feeds contained. And my script would be fetching a corrupted RSS feed and displaying it on my site, which could allow for cross-site scripting exploits to run riot.

So you need to write code as though the RSS feed is suspect, even if you trust the site that you are getting it from. This means doing what is necessary to avoid any third-party HTML making it into your page. The number of cross-site scripting exploits is high, so it can be difficult to know how secure your code is.

To reduce the number of exploits that can survive, my code calls a function named make_safe on everything that is going to be output in HTML.

function make_safe($string) {
    $string = preg_replace('#<!\[CDATA\[.*?\]\]>#s', '', $string);
    $string = strip_tags($string);
    // The next line requires PHP 5.2.3, unfortunately.
    //$string = htmlentities($string, ENT_QUOTES, 'UTF-8', false);
    // Instead, use this set of replacements in older versions of PHP.
    $string = str_replace('<', '&lt;', $string);
    $string = str_replace('>', '&gt;', $string);
    $string = str_replace('(', '&#40;', $string);
    $string = str_replace(')', '&#41;', $string);
    $string = str_replace('"', '&quot;', $string);
    $string = str_replace('\'', '&#039;', $string);
    return $string;
}

This function removes CDATA sections (see the note below), then calls PHP's strip_tags function to remove HTML markup, then tries to convert any remaining dangerous characters into HTML character entities (PHP 5 is better equipped to do this than PHP 4). However, if you see any holes still open to exploitation, tell me about it.

Embedded markup and CDATA sections

Some RSS feeds embed HTML markup in their title and description elements using HTML entities. These are harmless, but they will cause the news item to appear on your page surrounded by HTML tags. There are various ways to remove such unwanted markup, but each feed will require its own adjustments, so I don't offer suggestions here. [See February 2012 update below.]

Another issue is that some RSS feeds contain CDATA sections to hide non-text elements inside RSS elements, such as HTML markup. To get rid of these CDATA sections, my code deletes everything in them with the following code:

$string =  preg_replace('#<!\[CDATA\[.*?\]\]>#s', '', $string);

The RSS feed for dpreview.com only uses CDATA sections to store img elements inside description elements, but some RSS feeds seem to wrap everything in CDATA sections, even plain text. So, as with removing unwanted markup tags, you may need to craft a solution customised to the content of the feed you are using.

Other XML feed formats

The code in this guide has been based on parsing an RSS 2.0 feed. Some websites only offer Atom feeds for syndication. The Atom standard uses empty elements for some information, such as its link element. In such a case, you can use the element_attributes function to extract data from XML elements, and then the rest of the code suggested above can remain very similar.

The above methods aren't limited to syndication feeds. You can use the functions to parse data out of XML produced by website APIs. For instance, the Amazon Associates system returns results in XML, and you can use these functions to parse the data you want out of the XML.

Just bear in mind the limitations of using regex for XML parsing, and always consider the security of using data from external sources.

Updates

31 January, 2010: For anyone using PHP version 5.2.3 or later, I recommend you take a look at my article about reading XML feeds using SimpleXML, which shows you an easy way of constructing an object tree full of data from an XML document.

25 February, 2012: The dpreview.com feed has (for some time now) been placing HTML markup inside its description elements, which is why you might have seen img and p elements in the text of the news feed display at the top of this page recently. To remove these I've modified the make_safe function so that after the first call to strip_tags the resulting string is fed to html_entity_decode to convert entities back to raw characters, and then strip_tags is called again to remove any HTML tags which result. This seems to do the trick, but I've not got a full PHP development platform setup on my machine at the moment, so I've not been able to do any testing. Also, you need at least PHP 4.3.0 to use the html_entity_decode function so this won't suit some users (though as PHP 4 is no longer in support you really should have moved to PHP 5 by now).