Parsing the Xml returned from Twitter

The next step in this series is to parse the Xml data returned from the Twitter API into an array of status objects, then bind them to the table in our main window.

The article I referenced in the first post shows how to use the cocoa class NSXMLDocument and its friends to parse an xml document:
http://cocoawithlove.com/2008/09/cocoa-application-driven-by-http-data.html

Putting that together with the NSXMLElement docs at Apple:
http://developer.apple.com/DOCUMENTATION/Cocoa/Reference/Foundation/Classes/NSXMLElement_Class/Reference/Reference.html

And some XPath documentation courtesy of W3Schools:
http://www.w3schools.com/XPath/xpath_syntax.asp

We're armed with enough knowledge to start parsing the Twitter Xml response.

In our app we're currently mapping an array of Friend objects (first_name, last_name) to the table in the main window, so we can just as easily map an array of TwitterStatus objects to the table with very little change in code.

First we'll create a TwitterStatus class with accessors for the various fields. Although we'll only be dealing with the users' names and statuses in this article, we'll take the opportunity to grab the users' profile image while we're here for the next article.

class TwitterStatus
	attr_accessor :text, :screen_name, :profile_image_url
end

Then using the example from the Cocoa With Love article and the Apple docs as a starting point, we can write the following method for our TwitterApi class to parse the Xml returned from Twitter into an array of our TwitterStatus objects.

def parseXml(responseData)

	if not responseData then return end
	err = nil
	
	# Create an XMLDocument object from the Xml string
	begin
		document = NSXMLDocument.alloc.initWithData responseData, options:NSXMLDocumentTidyXML, error:err
	rescue StandardError => e
		alert(e.to_s)
		return
	end

	# Create arrays of all the elements we want extracting from the Xml using XPath
	rootNode = document.rootElement
	statusTexts = rootNode.nodesForXPath "//status/text", error:err
	userScreenNames = rootNode.nodesForXPath "//status/user/screen_name", error:err
	userProfileImages = rootNode.nodesForXPath "//status/user/profile_image_url", error:err

	result = []
	statusTexts.length.times do |i|
	
		# Iterate over every status and populate a new TwitterStatus object with them
		status = TwitterStatus.new
		status.text = statusTexts[i].stringValue
		status.screen_name = userScreenNames[i].stringValue
		status.profile_image_url = userProfileImages[i].stringValue
		result << status
	
	end
	
	return result
	
end

Next a quick change to the timeline method so we parse the Xml and hand the resulting array to the callback.

def getTimeline(callback)

	getUrl('http://twitter.com/statuses/friends_timeline.xml', 
            true, lambda {|data| callback.call(self.parseXml(data)) })

end

Which we can then consume from our refresh button like so:

@twitter.getTimeline(lambda do |data|

	@friends = data
	@friendsTableView.reloadData
	
end)

The final thing to do to make the whole thing work is to tweak the tableView method to return the correct properties from the TwitterStatus objects.

def tableView(view, objectValueForTableColumn:column, row:index)
	friend = @friends[index]
	case column.identifier
		when 'screen_name'
			friend.screen_name
		when 'text'
			friend.text
	end
end

Now hitting the refresh button should fill the table with your 20 most recent updates from Twitter. If it doesn't, then you've done something wrong. Don't blame me.

Someone asked in the comments of one of the previous articles if I could post the whole code or upload to github. I'll do that after I post the next article in about a week.

Speaking of which, the next article will show how to download the users' profile images and use them in the table instead of their name.

UPDATE: Part Three of this series is now available

Adding Basic Authentication

Adding Basic Authentication to a url request is pretty straight forward, in fact it only requires a header to be added to the request with the NSURLRequest.addValue:forHTTPHeaderField method. Simple huh? Not quite. The HTTP spec requires that the username and password be Base64 encoded, which is usually a case of requiring the base64 library in standard Ruby, but that isn't available in MacRuby, and there doesn't appear to be Cocoa object designed for the purpose.

After spending five minutes googling turned up exactly bot all, I decided to change tack and find out how to base64 encode myself.

How to Base64 Encode
http://email.about.com/cs/standards/a/base64_encoding.htm

So with a combination of the above article, a trusty Ruby book, blood, sweat, tears and a lot of trial and error I constructed:

class Base64

	ALPHA = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+-="

	def encode(str)
	
		dat = []
		str.each_byte { |b| dat << b }
		
		# Append 0s to the data to make it's length divisible by 3
		missingChars =  3 - (dat.length % 3)		
		missingChars.times { dat << 0 }
		
		result = ''
		sections = dat.length / 3
		sections.times do |i|
		
			a = dat[(i * 3) + 0] >> 2
			b = dat[(i * 3) + 0] << 4 & 63
			b = b | dat[(i * 3) + 1] >> 4
			c = dat[(i * 3) + 1] << 2 & 63 
			c = c | dat[(i * 3) + 2] >> 6
			d = dat[(i * 3) + 2] & 63
			
			if dat[(i * 3) + 1] == 0 then c = 64 end
			if dat[(i * 3) + 2] == 0 then d = 64 end
		
			result += ALPHA[a] + ALPHA[b] + ALPHA[c] + ALPHA[d]

		end

		return result
	end

end

There's no point me stepping through it, you should be able to dissect it easy enough with the referenced article, or just use it if you don't care how it works - it's not perfect and probably won't work in all scenarios, but it's good enough for Base64 encoding a username/password combination.

We can then use the Base64 class to encode our credentials and add the Basic Authentication header, which I've chosen to do in a separate method of the TwitterApi class called addAuthentication:

def addAuthentication(req)
	enc = Base64.new
	data = @username + ':' + @password
	data = enc.encode(data)
	data = 'Basic ' + data
	
	req.addValue data, forHTTPHeaderField:'Authorization'
end

I've also added a couple of attributes to the class that allow the username and password to be set:

attr_accessor :username, :password

And I've added a call to addAuthentication to the getUrl method as well as an authenticated parameter to the method signature, allowing you to specify if you want the request to be authenticated or not:

	def getUrl(urlString, authenticated, delegate)

		callback = HttpRequestCallback.new
		callback.delegate = delegate
		callback.buf = NSMutableData.new
		callback.response = nil

		url = NSURL.URLWithString(urlString)
		request = NSMutableURLRequest.requestWithURL(url, cachePolicy:POLICY, timeoutInterval:TIMEOUT)

		addAuthentication(req) if authenticated # <-- Add Authentication if required

		callback.conn = NSURLConnection.alloc.initWithRequest(request, delegate:callback)

	end

Finally I've added a getTimeline method to the class that calls getUrl for the authenticated user's timeline:

def getTimeline(callback)

	getUrl('http://twitter.com/statuses/friends_timeline.xml', true, lambda {|data| callback.call(data) })

end

We can consume this like so:

twitter = TwitterApi.new
twitter.username = 'xxx'
twitter.password = 'xxx'
twitter.getTimeline(lambda do |data|

	result = NSString.alloc.initWithData data, encoding:NSUTF8StringEncoding

	# display the result in an alert box
	alert = NSAlert.new
	alert.setMessageText(result)
	alert.runModal()	

end)

Which should download the Xml of your authenticated Twitter timeline and display it in a message box.

So we can now obtain our own timeline authenticated using Basic Authentication; next we'll parse the Xml into an array of status objects and bind them to the table in our main window. Part Three Available Here.

UPDATE: Part Two of this series is now available

Howdy. This is a new series I intend to write on how to create a Twitter client in MacRuby using the Cocoa framework on OSX. There doesn't seem to be a lot of articles around on this subject, so I thought it would be beneficial to the world at large if I chronicled my own journey.

Warning: this is not a "Learning Ruby" guide, if you don't know how to program in Ruby, or have a good Ruby book at hand, then you ain't gonna get too far with this.

I first started by following the original Getting Started with MacRuby tutorial provided by Apple, which, as well as giving you a good introduction to MacRuby, gets you to the starting point for this series - we'll adapt the application you build following that tutorial into our Twitter client, so off you go and follow that. We'll see you back here in half an hour.

---

Welcome back. Now that you've followed the Apple tutorial, you should have an application that contains a table and a button - the next step, and the purpose of this first article, is to add the ability to download data from a url.

There are easy ways to do synchronous url requests, such as NSString.initWithContentsOfURL, but synchronous calls would freeze our UI while we wait for the call to complete, which is bad for the user, so we need to asynchronously request our urls.

A bit of googling found me an article in Objective-C that taught me everything we need to do to achieve our goal, unfortunately it was in Objective-C:
http://cocoawithlove.com/2008/09/cocoa-application-driven-by-http-data.html

Luckily, after searching on the methods used in the Objective-C article, I managed to find some MacRuby code that implemented async url requests, so saved myself a big Cocoa->Ruby headache:
http://github.com/psychs/limechat/blob/4c8a75aff8f3f7af10bf1e6049baeee29af7d82b/ruby/lib/pasternakclient.rb

The NSURLConnection.initWithRequest method requires a delegate be provided that meets certain criteria. The request method will immediately return and the actual request will be performed in the background - once the request completes, it will hand the result to the code in this delegate object.

Both the Obj-C and MacRuby examples passed 'self' in as that delegate, but that would limit our object to only being able to run one request at a time. Therefore I knocked up the following class to be the delegate we pass, which allows us to set a delegate for each request:

class HttpRequestCallback
	attr_accessor :delegate, :buf, :response, :conn
	
	def cancel
		if @conn
			@conn.cancel
			@conn = nil
		end
	end
  
	def connection(conn, didReceiveResponse:res)
		return if @conn != conn
		@response = res
	end
 
	def connectionDidFinishLoading(conn)
		if @response
			code = @response.statusCode
			if code.to_s =~ /^20[01]$/
				@delegate.call(@buf)
			else
				@delegate.call(false)
			end
		end
		@conn = nil
	end
  
	def connection(conn, didReceiveData:data)
		return if @conn != conn
		@buf.appendData(data)
	end
  
	def connection(conn, didFailWithError:err)
		if @conn == conn
			@delegate.call(false)
		end
		@conn = nil
	end
  
	def connection(conn, willSendRequest:req, redirectResponse:res)
		return nil if @conn != conn
		if res && res.statusCode == 302
			@delegate.call(req.URL.to_s)
			@conn = nil
			nil
		else
			req
		end
	end	
end

The idea being that we create one of these each time we request a url, the request then communicates with this class which in turn calls a method of our choosing when the request is complete. Think of it like a middle man: We want the request to call one of our methods when it completes, however, the NSUrlConnection class requires a bit of infrastructure in place to deal with things, so this class is the middleman who has the complicated conversation with NSUrlConnection and then simply hands us an envelope with the answer.

With the HttpRequestCallback class in place, we can now implement the following getUrl method in a TwitterApi class:

class TwitterApi
	TIMEOUT = 10
	POLICY = NSURLRequestReloadIgnoringLocalCacheData

	def getUrl(urlString, delegate)

		callback = HttpRequestCallback.new
		callback.delegate = delegate
		callback.buf = NSMutableData.new
		callback.response = nil

		url = NSURL.URLWithString(urlString)
		request = NSMutableURLRequest.requestWithURL(url, cachePolicy:POLICY, timeoutInterval:TIMEOUT)
		callback.conn = NSURLConnection.alloc.initWithRequest(request, delegate:callback)

	end
end

We call this method passing it a url string and a lambda method that will be called when the request returns. The lambda will be called with one parameter of type NSMutableData, which we can convert into a string using NSString.initWithData. We can therefore consume and test our getUrl method by replacing the code in the addFriend method with:

def addFriend(sender)
	twitter = TwitterApi.new
	twitter.getUrl('www.google.com', lambda do |data|

		result = NSString.alloc.initWithData data, encoding:NSUTF8StringEncoding

		# display the result in an alert box
		alert = NSAlert.new
		alert.setMessageText(result)
		alert.runModal()	

	end)
end

Now when you run your application and click on the Add button, you should get a message alert containing the HTML source for the Google homepage.

Next Article: We need to be able to authenticate ourselves with the Twitter API in order to get access to our own Twitter stream - so we'll cover Basic Authentication and Base64 encoding. Part Two Available Here.

Further Reading:

Additional MacRuby Tutorial - makes understanding Objective-C syntax easier:
http://www.macruby.org/trac/wiki/MacRubyTutorial

Apple Objective-C Messaging:
https://developer.apple.com/documentation/Cocoa/Conceptual/ObjectiveC/Articles/chapter_2_section_4.html

Apple reference for NSUrlReq:
http://developer.apple.com/documentation/Cocoa/Reference/Foundation/Classes/NSMutableURLRequest_Class/Reference/Reference.html

Article on using NSUrlReq in Objective-C:
http://cocoawithlove.com/2008/09/cocoa-application-driven-by-http-data.html

The Twitter API Documentation:
http://apiwiki.twitter.com