Reading from the Tumblr API

I am a very keen cinema-goer and for the past few years I have kept a Tumblr blog of the films I have seen. There is only a very brief description of each film, along with a YouTube clip, but I am very keen to have a backup of the over 340 posts.

http://cinemacento.tumblr.com/

I am also not particularly keen on the Tumblr interface for writing the posts and have always felt that it would be much easier to use Umbraco and push the posts to Tumblr using their API. I could also then have the posts on my site and implement my own filters, etc.

Each of my posts has a particular structure, which I fudge into Tumblr. Here’s my process for creating a new film blog entry directly on Tumblr.

1. Find the trailer for the film on YouTube and copy the “Share” URL.
2. Log onto Tumblr and create a new “Text” blog entry.
3. Enter the film name as the title.
4. Select the special “video” widget in the text description and paste the YouTube URL in, to create the video clip in the blog entry.
5. Write my comments about the film after the video, in the description field.
6. Add any tags, typically the film name and sometimes some standard others that I use.
7. Save the entry, it will appear on the site.
8. Click on the settings button and “Edit” the entry.
9. Click on the (other) settings button to see the drop-down.
10. Change the “Post date” (in the annoying plain text format that they use).
11. Change the “Text editor” to be HTML.
12. Copy the star rating that I want to use from a text editor (each rating is a number of <div> HTML elements).
13. Paste the rating HTML into the beginning of the post.
14. Save the entry.

Umbraco Document Type

My first step was to create a document type in Umbraco that could hold the film blog entry details, along with an allowable folder structure to keep everything in order.

The star rating is implemented as an Umbraco Slider, the tags using the Umbraco Tags and there is an extra “Image” field for the few cases where I have used an image, rather than a video, for the film.

The following section details the code I used to read my posts from Tumblr and create their equivalents within my Umbraco structure.

Setup

I could not find any packages within Umbraco to help with the Tumblr API, so with the idea that I might create one in the future, I created a new project within my solution, in order to keep the API code independent of my website.

I also needed to include the Newtonsoft NuGet package in the separate project, in order to decode the JSON returned from the API.

The new project’s DLL must then be included in my website project, so that I can access the code. The reference manager is reached by clicking on "References" in the Solution Explorer in Visual Studio.

In order to use the Tumblr API you need to register an application on their site. I don’t think I’ll be using the “Callback URL”, but I put in a sensible URL that I could implement in the future if need be. I also kept a note of the consumer and secret keys for use later.

https://www.tumblr.com/oauth/apps

API Code

My first test call to the API was to use the “retrieve blog info” method of the API, this required providing the blog URL, which in my case is “cinemacento.tumblr.com”. Here’s the beginnings of the class that stores the consumer key (noted above) and can then be used to provide methods to access the API.

using System;
using System.Net;

namespace TumblrApi
{
    public class TumblrClient
    {
        private String apiKey;

        public TumblrClient(String oAuthConsumerKey)
        {
            apiKey = oAuthConsumerKey;
        }

        public String BlogInfo(String blogIdentifier)
        {
            WebClient client = new WebClient();
            String content = client.DownloadString("https://api.tumblr.com/v2/blog/" + blogIdentifier + "/info?api_key=" + apiKey);

            var obj = Newtonsoft.Json.JsonConvert.DeserializeObject<dynamic>(content);

            return obj.response.blog.description;
        }
    }
}

Further documentation on the Tumblr API can be found here.

https://www.tumblr.com/docs/en/api/v2

The function above can then be called using.

TumblrClient tumblrClient = new TumblrClient("{your-key-in-here}");
String description = tumblrClient.BlogInfo("cinemacento.tumblr.com");

As I want typed (rather than dynamic) objects being returned I need to create classes matching the structure of the Tumblr API responses.

namespace TumblrApi.Model
{
    public class BlogStatusAndResponse
    {
        public ReturnStatus meta { get; set; }
        public BlogResponse response { get; set; }
    }

    public class BlogResponse
    {
        public Blog blog { get; set; }
    }

    public class Blog
    {
        public Boolean ask { get; set; }
        public Boolean ask_anon { get; set; }
        public String ask_page_title { get; set; }
        public Boolean can_subscribe { get; set; }
        public String description { get; set; }
        public Boolean is_nsfw { get; set; }
        public int likes { get; set; }
        public String name { get; set; }
        public int posts { get; set; }
        public Boolean share_likes { get; set; }
        public Boolean subscribed { get; set; }
        public String title { get; set; }
        public int total_posts { get; set; }
        public int updated { get; set; }
        public String url { get; set; }
        public String uuid { get; set; }
        public Boolean is_optout_ads { get; set; }
    }
}

The “blog info” function was then rewritten as:

public Blog BlogInfo(String blogIdentifier)
{
	WebClient client = new WebClient();
	String content = client.DownloadString("https://api.tumblr.com/v2/blog/" + blogIdentifier + "/info?api_key=" + apiKey);

	BlogStatusAndResponse obj = Newtonsoft.Json.JsonConvert.DeserializeObject<BlogStatusAndResponse>(content);

	return obj.response.blog;
}

In a similar manner I was able to create a method to retrieve the posts from my blog. I have added a couple of extra parameters to this in order to control the “pages” of posts that are returned. Tumblr imposes a maximum of 20 at-a-time. Note the UTF8 encoding which Tumblr uses, this will stop any strange characters being returned.

public PostsResponse Posts(String blogIdentifier, int startPost=0, int numberOfPosts=20)
{
	WebClient client = new WebClient();
	client.Encoding = Encoding.UTF8;
	String content = client.DownloadString("https://api.tumblr.com/v2/blog/" + blogIdentifier + "/posts?api_key=" + apiKey 
										+ "&offset=" + startPost + "&limit=" + numberOfPosts);
	PostsStatusAndResponse posts = Newtonsoft.Json.JsonConvert.DeserializeObject<PostsStatusAndResponse>(content);
	return posts.response;
}

Using the API Code to Read Posts

The reading of my blog posts will be a one-off exercise, so in my website project I have implemented an UmbracoAuthorizedApiController to trigger it. The URL can only be accessed by someone who is logged into the back office. The code makes use of the HTML Agility Pack to read the HTML body of the post and extract the relevant information.

using HtmlAgilityPack;
using rtmullinger2018.Models;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using TumblrApi;
using TumblrApi.Model;
using Umbraco.Core.Models;
using Umbraco.Core.Services;
using Umbraco.Web.WebApi;

namespace rtmullinger2018.Controllers
{
    public class TumblrController : UmbracoAuthorizedApiController
    {
        // http://localhost:56226/umbraco/backoffice/api/Tumblr/GetTumblrPosts
        public String GetTumblrPosts()
        {
            String rtn = "<html><body>";
            int numberPostsCreated = 0;

            IPublishedContent cinemaCentoRoot = GetRoot();
            if (cinemaCentoRoot == null)
            {
                rtn += "<p>Can't find Cinema Cento root.</p></body></html>";
                return rtn;
            }

            TumblrClient tumblrClient = new TumblrClient("{api-key-here}");
            Blog blog = tumblrClient.BlogInfo("cinemacento.tumblr.com");

            int firstPostInPage = 0;
            int postsInPage = 20;

            while (firstPostInPage < blog.posts)
            {
                PostsResponse posts = tumblrClient.Posts("cinemacento.tumblr.com", firstPostInPage, postsInPage);

                foreach (Post post in posts.posts)
                {
                    // Re-get the root, as it may now have new "year" children.
                    cinemaCentoRoot = GetRoot();

                    IPublishedContent parentFolder = GetParentFolder(cinemaCentoRoot, post);
                    if (parentFolder == null)
                    {
                        rtn += "<p>Could not find or create a parent folder for post " + post.title + "; " + post.date + "</p>";
                    }

                    String message = CreateNewNode(parentFolder, post);

                    if (!String.IsNullOrWhiteSpace(message))
                    {
                        rtn += "<p>" + message + "</p>";
                    }
                    else
                    {
                        numberPostsCreated++;
                    }
                }

                firstPostInPage += postsInPage;
            }

            rtn += "<p>Number posts created: " + numberPostsCreated + "</p></body></html>";

            return rtn;
        }

        private String CreateNewNode(IPublishedContent parentFolder, Post post)
        {
            String rtn = "";

            CinemaCentoPostHtml htmlInfo = GetHtmlInfo(post);

            String tags = "";
            foreach (String tag in post.tags)
            {
                if (!String.IsNullOrWhiteSpace(tags))
                {
                    tags += ",";
                }

                tags += tag;
            }

            IContentService contentService = ApplicationContext.Services.ContentService;
            IContent newNode = contentService.CreateContent(post.title, parentFolder.Id, "cinemaCento");
            newNode.SetValue("title", post.title);
            newNode.SetValue("description", htmlInfo.Text);
            newNode.SetValue("videoURL", htmlInfo.YouTubeUrl);
            newNode.SetValue("starRating", htmlInfo.StarRating);
            newNode.SetValue("date", post.date);
            newNode.SetValue("tags", tags);

            var result = contentService.SaveAndPublishWithStatus(newNode);

            if (!result.Success)
            {
                rtn = "Error creating node: " + result.Exception.Message;
            }

            return rtn;
        }

        private CinemaCentoPostHtml GetHtmlInfo(Post post)
        {
            CinemaCentoPostHtml html = new CinemaCentoPostHtml();

            HtmlDocument htmlDoc = new HtmlDocument();
            htmlDoc.LoadHtml(post.body);

            HtmlNode starRatingDiv = htmlDoc.DocumentNode.SelectSingleNode("//div");
            if (starRatingDiv != null)
            {
                html.StarRating = 0.0f;
                HtmlNodeCollection stars = starRatingDiv.SelectNodes("//i");
                foreach (HtmlNode star in stars)
                {
                    HtmlAttribute starClass = star.Attributes["class"];
                    if (starClass != null && !String.IsNullOrWhiteSpace(starClass.Value))
                    {
                        String[] classes = starClass.Value.Split(' ');
                        if (classes.Contains("fa-star"))
                        {
                            html.StarRating += 1.0f;
                        }
                        else if (classes.Contains("fa-star-half-o"))
                        {
                            html.StarRating += 0.5f;
                        }
                    }
                }
            }

            HtmlNode figure = htmlDoc.DocumentNode.SelectSingleNode("//figure");
            if (figure != null)
            {
                HtmlAttribute figureUrl = figure.Attributes["data-url"];
                if (figureUrl != null && !String.IsNullOrWhiteSpace(figureUrl.Value))
                {
                    html.YouTubeUrl = HttpUtility.UrlDecode(figureUrl.Value);
                }
            }

            HtmlNode textParagraph = htmlDoc.DocumentNode.SelectSingleNode("//p");
            if (textParagraph != null && !String.IsNullOrWhiteSpace(textParagraph.InnerText))
            {
                html.Text = textParagraph.InnerText;
            }

            return html;
        }

        private IPublishedContent GetParentFolder(IPublishedContent root, Post post)
        {
            IContentService contentService = ApplicationContext.Services.ContentService;
            IPublishedContent monthFolder = null;

            String year = post.date.ToString("yyyy");
            String month = post.date.ToString("MMMM");

            IPublishedContent yearFolder = root.Children.Where(x => x.DocumentTypeAlias == "cinemaCentoFolder" && x.Name == year).FirstOrDefault();
            if (yearFolder == null)
            {
                IContent newNode = contentService.CreateContent(year, root.Id, "cinemaCentoFolder");
                var result = contentService.SaveAndPublishWithStatus(newNode);
                if (!result.Success)
                {
                    return null;
                }

                root = Umbraco.TypedContent(root.Id);    // Re-fetch to get new child
                yearFolder = root.Children.Where(x => x.DocumentTypeAlias == "cinemaCentoFolder" && x.Name == year).FirstOrDefault();
            }

            if (yearFolder != null)
            {
                monthFolder = yearFolder.Children.Where(x => x.DocumentTypeAlias == "cinemaCentoFolder" && x.Name == month).FirstOrDefault();
                if (monthFolder == null)
                {
                    IContent newNode = contentService.CreateContent(month, yearFolder.Id, "cinemaCentoFolder");
                    var result = contentService.SaveAndPublishWithStatus(newNode);
                    if (!result.Success)
                    {
                        return null;
                    }

                    yearFolder = Umbraco.TypedContent(yearFolder.Id);    // Re-fetch to get new child
                    monthFolder = yearFolder.Children.Where(x => x.DocumentTypeAlias == "cinemaCentoFolder" && x.Name == month).FirstOrDefault();
                }
            }

            return monthFolder;
        }

        private IPublishedContent GetRoot()
        {
            IPublishedContent cinemaCentoRoot = null;
            IPublishedContent dataRoot = Umbraco.TypedContentAtRoot().Where(x => x.DocumentTypeAlias == "dataFolder").FirstOrDefault();
            if (dataRoot != null)
            {
                cinemaCentoRoot = dataRoot.Children.Where(x => x.DocumentTypeAlias == "cinemaCentoFolder").FirstOrDefault();
            }

            return cinemaCentoRoot;
        }
    }
}

The code requires a poco class to represent the HTML body of the post.

namespace rtmullinger2018.Models
{
    public class CinemaCentoPostHtml
    {
        public float StarRating { get; set; }
        public String YouTubeUrl { get; set; }
        public String Text { get; set; }
    }
}