Getting data from XML feeds using PHP

A how-to guide by Bobulous.

Introduction

[Note: This article has been superseded by the newer article reading XML feeds using SimpleXML.]

In a previous article, I introduced a set of simple functions that grab data from XML files. The functions are a little rough and ready, but with well-structured XML data formats they work very well. This article offers an example of how to use those functions to grab the latest headlines from a real-world RSS feed and then format them into a "latest news" box for your website.

The desired result

Here is the result of fetching an XML feed, extracting the desired data from it, and formatting that data into a block of HTML. Using CSS, you can alter the appearance of the news box radically, choosing colours, sizes, positions and fonts to suit the situation.

Articles: Digital Photography Review (dpreview.com)

Cold War camera: 1950s Berlin in color (part 1)
Ten years after the end of WWII, Germany was rebuilding. Cities like Berlin, which had been severely damaged during the war were emerging from the rubble as the 'Wirtschaftswunder' or 'economic miracle' transformed West...
[09:00, 1st September GMT]
Readers' Showcase: Black and White photography
While it's easy to get swept up in the stream of product announcements at this time of year, it's also a good idea to remember why we buy that gear - to make great photos. What better way to do it than to showcase the excellent...
[09:00, 31st August GMT]
Nikon releases D4S firmware C:1.10
Nikon has issued a firmware update for the D4S. The long list of improvements included in this firmware version are mostly minor bug fixes and tweaks to menus and camera operation. See the full details and get a link to download the latest...
[09:00, 30th August GMT]
Sony adds XAVC S support to Cyber-shot DSC-RX10
The Sony Cyber-shot DSC-RX10 is a video powerhouse, and Sony is now giving it the same XAVC S codec found on its RX100 III and a5100 models via firmware update v.2. This allows for 1080p/60/30/24 and 720p/120 resolutions at a bit rate of...
[22:53, 29th August GMT]
Week in review: 8 things we learned
It's been a busy week and as we head into a long weekend here in the US, it seemed like a good time to take a look back at everything that's happened in the past seven(ish) days. With new products from Fujifilm, Olympus and...
[22:01, 29th August GMT]

Feed fetched 2014-09-02 15:23:08 UTC

Step one: Get the RSS feed

Before you create a script that fetches an XML feed from someone else's site, make sure to check that you have permission to do so. Look for a usage policy in the feeds section of the site, or email the site owner to make sure they're happy with what you intend to do.

For this article, I'm using the latest news feed from Digital Photography Review, one of my favourite sites. The XML feeds at dpreview.com are in the RSS 2.0 format, but the techniques used in this article could be applied to most XML data formats.

Once you have the URL of the XML feed that you are going to use, you need to have PHP load the contents of the feed into a string variable. Using file_get_contents, you could fetch the XML file like so:

$xml = file_get_contents('http://www.dpreview.com/feeds/news.xml');

However, this requires that PHP is setup with allow_url_fopen set to true. Not all web hosts enable this setting, for security reasons. So another way to fetch the XML file into a string is by using the cURL functions (if they are installed on your PHP setup) like this:

// Use cURL to get the RSS feed into a PHP string variable.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,
        'http://www.dpreview.com/feeds/news.xml');
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$xml = curl_exec($ch);
curl_close($ch);

Either method should give you a string called $xml that contains the contents of the entire XML feed.

Step two: Extract data from the feed

Most XML data files contain a large amount of content. For instance, the dpreview.com news feed contains channel (feed) title and image information, a couple of dozen news items, each with a title, description, image for the item, link to visitor comments, and the publication date. To keep things simple, I'm only going to extract the channel title and URL, and the title, description and date of each news item.

Starting off with the channel details, we can grab what we want using the value_in function. (You need to include the functions in xml_regex.php, which you can download from my page about extracting XML data with regular expressions.)

// Include the handy XML data extraction functions.
include 'xml_regex.php';
// An RSS 2.0 feed must have a channel title, and it will
// come before the news items. So it's safe to grab the
// first title element and assume that it's the channel
// title.
$channel_title = value_in('title', $xml);
// An RSS 2.0 feed must also have a link element that
// points to the site that the feed came from.
$channel_link = value_in('link', $xml);

Next we want to build an array that contains individual news item elements. This can be done using the element_set function.

// Create an array of item elements from the XML feed.
$news_items = element_set('item', $xml);

Once we've got the array of item elements, we can iterate through the array, one item at a time, and extract just the data that we're interested in from each news item: the URL, title, description and date. Then we can store that data in a new array called $item_array.

foreach($news_items as $item) {
    $title = value_in('title', $item);
    $url = value_in('link', $item);
    $description = value_in('description', $item);
    $timestamp = strtotime(value_in('pubDate', $item));
    $item_array[] = array(
            'title' => $title,
            'url' => $url,
            'description' => $description,
            'timestamp' => $timestamp
    );
}

Now we have $item_array, which is an array that contains news items in the form of associative arrays, so that the title of the first news item should be found in $item_array[0]['title'] and the title of the second news item should be found in $item_array[1]['title'] and so on.

Step three: Markup the data with HTML

With all the data we want stored in one handy array, we can iterate through the array and markup the data with HTML.

if (sizeof($item_array) > 0) {
    // First create a div element as a container for the whole
    // thing. This makes CSS styling easier.
    $html = '<div class="rss_feed_headlines">';
    // Markup the title of the channel as a hyperlink.
    $html .= '<h2 class="channel_title">'.
            '<a href="'.make_safe($channel_link).'">'.
            make_safe($channel_title).'</a></h2><dl>';
    // Now iterate through the data array, building HTML for
    // each news item.
    $count = 0;
    foreach ($item_array as $item) {
        $html .= '<dt><a href="'.make_safe($item['url']).'">'.
                make_safe($item['title']).'</a></dt>';
        $html .= '<dd>'.make_safe($item['description']);
        if ($item['timestamp'] != false) {
		    $html .= '<br />' .
                    '<span class="news_date">['.
                    gmdate('H:i, jS F T', $item['timestamp']).
                    ']</span>';
        }
        echo '</dd>';
        // Limit the output to five news items.
        if (++$count == 5) {
            break;
        }
    }
    $html .= '</dl></div>';
    echo $html;
}

The above code creates the HTML for the first five news items, plus the channel title, and wraps it all in a div element with the class "rss_feed_headlines" so that CSS stylesheets can target the news box specifically.

Step four: Consider security implications of external data

I trust the people at dpreview.com not to send anything malicious to me via their RSS feeds. But suppose their site was hacked or infected by malware. Then they wouldn't be in control of what their RSS feeds contained. And my script would be fetching a corrupted RSS feed and displaying it on my site, which could allow for cross-site scripting exploits to run riot.

So you need to write code as though the RSS feed is suspect, even if you trust the site that you are getting it from. This means doing what is necessary to avoid any third-party HTML making it into your page. The number of cross-site scripting exploits is high, so it can be difficult to know how secure your code is.

To reduce the number of exploits that can survive, my code calls a function named make_safe on everything that is going to be output in HTML.

function make_safe($string) {
    $string = preg_replace('#<!\[CDATA\[.*?\]\]>#s', '', $string);
    $string = strip_tags($string);
    // The next line requires PHP 5.2.3, unfortunately.
    //$string = htmlentities($string, ENT_QUOTES, 'UTF-8', false);
    // Instead, use this set of replacements in older versions of PHP.
    $string = str_replace('<', '&lt;', $string);
    $string = str_replace('>', '&gt;', $string);
    $string = str_replace('(', '&#40;', $string);
    $string = str_replace(')', '&#41;', $string);
    $string = str_replace('"', '&quot;', $string);
    $string = str_replace('\'', '&#039;', $string);
    return $string;
}

This function removes CDATA sections (see the note below), then calls PHP's strip_tags function to remove HTML markup, then tries to convert any remaining dangerous characters into HTML character entities (PHP 5 is better equipped to do this than PHP 4). However, if you see any holes still open to exploitation, tell me about it.

Embedded markup and CDATA sections

Some RSS feeds embed HTML markup in their title and description elements using HTML entities. These are harmless, but they will cause the news item to appear on your page surrounded by HTML tags. There are various ways to remove such unwanted markup, but each feed will require its own adjustments, so I don't offer suggestions here. [See February 2012 update below.]

Another issue is that some RSS feeds contain CDATA sections to hide non-text elements inside RSS elements, such as HTML markup. To get rid of these CDATA sections, my code deletes everything in them with the following code:

$string =  preg_replace('#<!\[CDATA\[.*?\]\]>#s', '', $string);

The RSS feed for dpreview.com only uses CDATA sections to store img elements inside description elements, but some RSS feeds seem to wrap everything in CDATA sections, even plain text. So, as with removing unwanted markup tags, you may need to craft a solution customised to the content of the feed you are using.

Other XML feed formats

The code in this guide has been based on parsing an RSS 2.0 feed. Some websites only offer Atom feeds for syndication. The Atom standard uses empty elements for some information, such as its link element. In such a case, you can use the element_attributes function to extract data from XML elements, and then the rest of the code suggested above can remain very similar.

The above methods aren't limited to syndication feeds. You can use the functions to parse data out of XML produced by website APIs. For instance, the Amazon Associates system returns results in XML, and you can use these functions to parse the data you want out of the XML.

Just bear in mind the limitations of using regex for XML parsing, and always consider the security of using data from external sources.

Updates

31 January, 2010: For anyone using PHP version 5.2.3 or later, I recommend you take a look at my article about reading XML feeds using SimpleXML, which shows you an easy way of constructing an object tree full of data from an XML document.

25 February, 2012: The dpreview.com feed has (for some time now) been placing HTML markup inside its description elements, which is why you might have seen img and p elements in the text of the news feed display at the top of this page recently. To remove these I've modified the make_safe function so that after the first call to strip_tags the resulting string is fed to html_entity_decode to convert entities back to raw characters, and then strip_tags is called again to remove any HTML tags which result. This seems to do the trick, but I've not got a full PHP development platform setup on my machine at the moment, so I've not been able to do any testing. Also, you need at least PHP 4.3.0 to use the html_entity_decode function so this won't suit some users (though as PHP 4 is no longer in support you really should have moved to PHP 5 by now).