I've back dated a load of the updates, as I'd been meaning to post blog entries about the work, but hadn't gotten round to it.

So far I have a 3d design worked out a working prototype circuit and a full breadboard circuit also designed.

I'll try and put up some full photo's soon and maybe even some video of the prototype.

We noticed at Yahoo! that we were started to get lots of timeouts from Facebook's APIs that we use on the Yahoo! homepage.

I've not heard anything official from Facebook yet, so this next part is all speculation.

I suspect that as a user updates their privacy settings, as forced to the first time they hit facebook.com now, Facebook flags that users account is dirty (something changed) and purges the cache of the users stream data.
Now that the cache is empty for a particular user, when a friend of that user views Facebook, the backend servers must go back to the master data source and confirm permissions on what is visible before it is displayed.

That is fine when you have a few users updating their permissions, if however you force ALL your users to do this, you site rapidly becomes overwhelmed, as you effectively removed your cache. This effect is known as "The Thundering Herd".

As the load increases, due to all extra requests for data that would normally be handled by caches, you start to get race conditions as requests return and try to populate the cache. Also as the load increases, requests start to take longer as connections are kept open while backends are queried. Because connections are being kept open, the server stops handling requests, as there are no more connection slots and you start to get errors as clients can no longer connect.

How do you prevent this. Don't push changes that require you whole userbase to update their records at once. Stage the release, starting with 1% until you can guage how much load it will generate on your servers. Then you can safely ramp up your change safely.

Which is good, as I'm doing a talk at SkillsMatter in December on that very subject.

Since I won an iPhone in a raffle last month I now have a surplus ipaq that used to be my portable brain, randomly I came across this site on running Linux on ipaqs. So this maybe the way to go.

However this guy also made some really cool bots running windows C++ code. I think certainly with an ipaq as the brains plus computer junk from the loft I probably have the kit for some interesting junkbots.

One other thing you'll notice is that, like my reading habits, I tend to have lots of projects on the go at the same time. I'll come back to them every now and again, and then may drop them for lengthy periods (sometimes years) which frustrates my wife who tells me she is a "Completer Finisher" mentality.

That probably makes me a mix of  Plant/Implementer/Specialist

Tesco is starting the rollout of it's new backend Grocery application, code named Martini (anytime, anyplace, anywhere).
They have opened up their new APIs from their grocery service to a few select developers. In fact very select, you had to attend to launch/briefing in person to register for the scheme.

A lot of time had obviously gone into preparing the event. With almost the whole conference room covered in brainstorm sheets and post it notes. These covered 5 main areas for potential developers to concentrate on or consider for their application. This was based on market and socioethnic research (hey that may not be the correct word but I didn't write it down and I'm now on the train).

After an initial introduction, we were all invited to chow down on pizza, lager and wine (actually nice for change to get red and white wine available) and browse the brainstormings, only slightly hindered by Microsoft's insistence that no one take food or drink into their meeting rooms.

The APIs look reasonably well thought out and the new Martini release adds some great new functionality, such as nutrition information and cheaper alternatives to products in your shopping basket.

There were some concerns voiced by some developers present at the briefing, quite rightly, over their security implementation, which requires the API consumer to forward the users login credentials, rather than implementing OAuth or another signature/token based authentication mechanism.
It looks like this will probably be addressed in a future release, once Martini is live and running Tesco's existing shopping site.
Customer basket checkout has yet to) be finalised, due to financial and legal compliance.

Everyone who attended the event and wishes to participate in the affliate scheme will get signed up to TradeDoubler and will earn £5 for each new customer they manage to entice into signing up to Tesco's online shopping service.
The tricky part of this is then the ongoing revenue.
For each basket purchased over the value of £50 the application developer gets a potential £0.10. Potential, as this is the proportional to the number of products added to basket by that particular application versus the total number of products in the basket.
Note also products in basket not items. 10 one penny chews would earn you the same as someone adding an iPhone or flatscreen tv. This is deliberate, to prevent skewing of apps towards higher value goods.

Having spent so long working on commercially sensitive applications, which is now in the open.

Anyway, apart from helping the Yahoo! OpenHack day, I also signed up to Tesco's T-JAM event which certainly looks like it could be interesting.

Tesco's API is a SOAP API, written in and for C# primarily, so it seems that some people had some issues getting started in PHP.

There is a revised API in the pipeline and also the addition of a REST API as well.

But here are some handy hints for getting started:

First off, PHP 5+ has a SOAP client (and server) built in, so here's a quick script that generates a valid login to the TESCO dev API.

<?php
define('SOAPSERVER', 'http://www.lansley.com/TescoAPI/TescoAPI.svc?wsdl');
define('DEV_KEY', 'your dev key');
define('APP_KEY', 'your app key');

$user  = 'email address of test user';
$pass  = 'password of test user';
$tesco = new SoapClient(SOAPSERVER);

$params = array(
                'email'          => $user,
                'password'       => $pass,
                'developerKey'   => DEV_KEY,
                'applicationKey' => APP_KEY
              );
$sess = $tesco->__soapCall('Login', array('parameters' => $params));
print_r($sess);

Which generates the following output:

stdClass Object
(
    [LoginResult] => stdClass Object
        (
            [StatusCode] => 0
            [StatusInfo] =>
        )

    [session] => stdClass Object
        (
            [ApplicationKey] => your app key
            [BranchNumber] => 2055
            [Cookies] => stdClass Object
                (
                    [string] => Array
                        (
                            [0] => sessionTest=
                            [1] => v=WggAACp8E%3d%3d
                            [2] => CustomerId=9xxx9
                            [3] => CID=93
                            [4] => BTCCMS=h1qzwp%c%3d
                            [5] => UIMode=00000
                            [6] => PS=CSC=D
                            [7] => SSVars=CustomerName=Bob&basketTotal=&basketCount=&slotStart=&slotEnd=&addrName=
                            [8] => u=oQgttVAeDK%2bg%3d%3d
                            [9] => t=dgcAA28XbJWE%3d
                        )

                )

            [CustomerName] => Bob
            [DeveloperKey] => your dev key
        )

)

I've monkeyed around with the values in some of the keys (as these are my login credentials) but you can see the general structure. The session object needs to be stored and used in subsequent requests.

To interrogate the API and view the functions and types available, there are some PHP functions to help:

<?php
define('SOAPSERVER', 'http://www.lansley.com/TescoAPI/TescoAPI.svc?wsdl');
define('DEV_KEY', 'your dev key');
define('APP_KEY', 'your app key');

$tesco = new SoapClient(SOAPSERVER);

print_r($tesco->__getFunctions());

Which shows the following SOAP methods available:

Array
(
    [0] => LoginResponse Login(Login $parameters)
    [1] => LogEventResponse LogEvent(LogEvent $parameters)
    [2] => ProductSearchResponse ProductSearch(ProductSearch $parameters)
    [3] => ListProductOffersResponse ListProductOffers(ListProductOffers $parameters)
    [4] => ListProductCategoriesResponse ListProductCategories(ListProductCategories $parameters)
    [5] => ListProductsByCategoryResponse ListProductsByCategory(ListProductsByCategory $parameters)
    [6] => SetProductRatingResponse SetProductRating(SetProductRating $parameters)
    [7] => GetProductRatingResponse GetProductRating(GetProductRating $parameters)
    [8] => GetMultipleProductRatingsResponse GetMultipleProductRatings(GetMultipleProductRatings $parameters)
    [9] => ListBasketItemsResponse ListBasketItems(ListBasketItems $parameters)
    [10] => AddItemToBasketResponse AddItemToBasket(AddItemToBasket $parameters)
    [11] => ChangeItemQuantityResponse ChangeItemQuantity(ChangeItemQuantity $parameters)
    [12] => DeleteItemFromBasketResponse DeleteItemFromBasket(DeleteItemFromBasket $parameters)
)

And:

<?php
define('SOAPSERVER', 'http://www.lansley.com/TescoAPI/TescoAPI.svc?wsdl');
define('DEV_KEY', 'your dev key');
define('APP_KEY', 'your app key');

$tesco = new SoapClient(SOAPSERVER);

print_r($tesco->__getTypes());

Which shows the following types available:

Array
(
    [0] => struct ArrayOfstring {
 string string;
}
    [1] => struct ResponseStatus {
 int StatusCode;
 string StatusInfo;
}
    [2] => struct Session {
 string ApplicationKey;
 string BranchNumber;
 ArrayOfstring Cookies;
 string CustomerName;
 string DeveloperKey;
}
    [3] => struct ArrayOfProduct {
 Product Product;
}
    [4] => struct Product {
 string Id;
 string ImagePath;
 string Name;
 string OfferPromotion;
 string OfferValidity;
 double Price;
 int Quantity;
 float Rating;
}
    [5] => struct ArrayOfCategory {
 Category Category;
}
    [6] => struct Category {
 ArrayOfCategory Children;
 string Href;
 string Id;
 string Name;
 string RawHtml;
}
    [7] => struct ProductRating {
 string Id;
 int MyRating;
 float Rating;
 int Rating1Count;
 int Rating2Count;
 int Rating3Count;
 int Rating4Count;
 int Rating5Count;
}
    [8] => struct Login {
 string email;
 string password;
 string developerKey;
 string applicationKey;
}
    [9] => struct LoginResponse {
 ResponseStatus LoginResult;
 Session session;
}
    [10] => struct LogEvent {
 Session session;
 string eventText;
}
    [11] => struct LogEventResponse {
 ResponseStatus LogEventResult;
}
    [12] => int char
    [13] => duration duration
    [14] => string guid
    [15] => struct ProductSearch {
 Session session;
 string searchString;
 boolean getRatings;
}
    [16] => struct ProductSearchResponse {
 ResponseStatus ProductSearchResult;
 ArrayOfProduct products;
}
    [17] => struct ListProductOffers {
 Session session;
 boolean getRatings;
}
    [18] => struct ListProductOffersResponse {
 ResponseStatus ListProductOffersResult;
 ArrayOfProduct offers;
}
    [19] => struct ListProductCategories {
 Session session;
}
    [20] => struct ListProductCategoriesResponse {
 ResponseStatus ListProductCategoriesResult;
 ArrayOfCategory categories;
}
    [21] => struct ListProductsByCategory {
 Session session;
 Category shelf;
 boolean getRatings;
}
    [22] => struct ListProductsByCategoryResponse {
 ResponseStatus ListProductsByCategoryResult;
 ArrayOfProduct products;
}
    [23] => struct SetProductRating {
 Session session;
 Product product;
}
    [24] => struct SetProductRatingResponse {
 ResponseStatus SetProductRatingResult;
}
    [25] => struct GetProductRating {
 Session session;
 Product thisproduct;
}
    [26] => struct GetProductRatingResponse {
 ResponseStatus GetProductRatingResult;
 ProductRating productrating;
}
    [27] => struct GetMultipleProductRatings {
 Session session;
 string spaceDelimitedProductIDs;
}
    [28] => struct GetMultipleProductRatingsResponse {
 ResponseStatus GetMultipleProductRatingsResult;
 string productRatingsInfo;
}
    [29] => struct ListBasketItems {
 Session session;
}
    [30] => struct ListBasketItemsResponse {
 ResponseStatus ListBasketItemsResult;
 ArrayOfProduct products;
}
    [31] => struct AddItemToBasket {
 Session session;
 Product item;
}
    [32] => struct AddItemToBasketResponse {
 ResponseStatus AddItemToBasketResult;
}
    [33] => struct ChangeItemQuantity {
 Session session;
 Product item;
}
    [34] => struct ChangeItemQuantityResponse {
 ResponseStatus ChangeItemQuantityResult;
}
    [35] => struct DeleteItemFromBasket {
 Session session;
 Product item;
}
    [36] => struct DeleteItemFromBasketResponse {
 ResponseStatus DeleteItemFromBasketResult;
}
)

Hope this helps, not just with the Tesco Grocery API, but in general debugging SOAP calls from within PHP.

Last week we turned on the opt in to the Yahoo! Homepage http://uk.yahoo.com/trynew

My personal project was writing the Facebook App, which does seem to have received a fair amount of press coverage.

However for the last 18 months we've been working on a complete rewrite of the homepage.

This has included releasing four countries (France, United Kingdom, United States and India) all using the same codebase and the same developers.

Personally I've been involved in setting up testing frameworks, continuous integration and code quality.

I'll try and publish further updates later. Specifically, if possible, information on the Facebook App.

This is a brief reminder of the series and a list of all the articles.

These articles are the originals and are being used as a base for a YDN article to be published shortly.

1. Introduction to CodeSniffer
2. CodeSniffer Output
3. Writing an example CodeSniffer Standard
4. How does CodeSniffer work?
5. Writing an example CodeSniffer Sniff

In the last post we looked at the internals of CodeSniffer and how the processing of files is done. This time we will look at how an individual Sniff is constructed.

Sniff Basics

• Each Sniff is a PHP class.
• Each class consist of a minimum of two methods.
  • A register method, that returns an array of tokens that the Sniff is interested in.
  • And a process method that is executed each time that token is encountered.
• Each Sniff class lives inside a Standard.
  • Each Sniff lives inside a subfolder "Sniffs" under the standard
  • Each Sniff lives inside a subfolder of "Sniffs" containing themed Sniffs
• Each Sniff class must adhere to the naming convention.

Starting a Sniff

Let's take a look at an example Sniff. For instance a Sniff designed to check that our PHP code has no extraneous whitespace outside of our PHP code, that could cause a PHP error should we subsequently send headers to the browser.

First of all we create a class and decide on a name for it.

As CodeSniffer uses an autoloader that loads the class' file and path based on the class name we have to stick to a specific naming convention.

class KingKludge_Sniffs_PHP_NoExtraneousWhiteSpaceSniff
implements PHP_CodeSniffer_Sniff {}

equates to a path of:

{CodeSniffer}/Standards/KingKludge/Sniffs/PHP/NoExtraneousWhiteSpaceSniff.php

where {CodeSniffer} is your CodeSniffer install path.

Secondly, inside our new class we must define our register method:

public function register()
{
	return array(
		T_OPEN_TAG,
		T_CLOSE_TAG,
		);
} 

As we want to check for whitespace before or after our PHP declarations, we state that this sniff is interested in PHP open and close tags.

Thirdly, we must define our process method:

public function process(PHP_CodeSniffer_File $phpcsFile, $stackPtr)
{

}

As you can see, our process method is passed some parameters.

These are $phpcsFile which is PHP_CodeSniffer_File object, containing the file being processed and $stackPtr an integer value pointing to CodeSniffer's current place in the file.

View the empty standard file on GitHub

Now on to the actually code in our Sniff.

As our proposed Sniff is going to look for whitespace being output before our page is ready to be rendered, we are interested in whitespace that occurs before the opening PHP tag or after the closing PHP tag.

We are only interested in whitespace, any HTML or other markup is okay.

So we can see if our pointer is at the beginning of the file, if so, drop out of the process.

$tokens = $phpcsFile->getTokens();
if (T_OPEN_TAG === $tokens[$stackPtr]['code'])
{
	if (0 === $stackPtr)
	{
		return;
	}
}

The getTokens() command returns all of the tokens for the current file in an associative array. (in case you didn't realise).

The rest should be self explanatory.

We can also do a similar check for a PHP close tag

if (T_CLOSE_TAG === $tokens[$stackPtr]['code'])
{
	$content = $tokens[$stackPtr]['content'];
	if (
	    false === isset($tokens[($stackPtr + 1)]) &&
	    trim($content) === $content
	   )
	{
		return;
	}
}

View the basic standard file on GitHub

This article will try to cover in-depth how CodeSniffer actually works to give insight into the next proposed article, writing a sniff from scratch.

The PHP Tokenizer

CodeSniffer works by extending the PHP tokenizer function.

Given the following section of code:

<?php

function DoSomething(array $foo) {
    print_r($foo);
}

?>   

and running it through PHP's native tokenizer we get the following output

PHP Tokenized version of foo.php

PHP's tokenizer only identifies a limited subset of PHP syntax as listed here.

All other potential tokens get either identifies as a string, signified by the sub array[0] integer value of 307 or the constant T_STRING, else it simply returns the string value of the token i.e. those seen array values 17,18 and 20 in the sample output above.

To map the above integer values to PHP's string constant name you can use the PHP function token_name()

For example:

$ php -r "print(token_name(369));"
T_CLOSE_TAG

As once the PHP tokenizer has run, we have a lot of code still encapsulated as T_STRING or with no tokenizing done, CodeSniffer takes these simple tokens and expands them further.

CodeSniffer introduces new constants such as T_TRUE, T_FALSE, T_NULL, T_PARENT, T_OPEN_CURLY_BRACKET and so on.

This gives CodeSniffer considerable scope to be able to handle much finer detail of the PHP syntax.

The PHP CodeSniffer Tokenizer

When CodeSniffer first loads, the standard in use is determined from the command line or from the stored config. The standard is then loaded and all of that standard's sniffs are loaded.

Each of the sniffs gets called via the register() method and a hash of all the tokens and classes is created.

Then CodeSniffer starts looking for the files to check, if a directory is specified, CodeSniffer iterates through the directories until a file with the correct extension is found, then each file is processed in turn.

If a list of files or a single file is specified, then the above step is skipped and CodeSniffer starts parsing the file(s) as defined in the parameters.

Once CodeSniffer has tokenized the file under analysis into one (rather large) multidimensional array of language syntax tokens, the rest is quite simple.

CodeSniffer breaks each file under examination down and does a series of context checks before processing the tokens and calling all the registered sniffs.

These checks are:

1. Bracket Map: checking braces
2. Scope Map: checking for class, function and conditional statement scopes
3. Level Map: checking for class, function and conditional statement levels

If we look at the CodeSniffer Tokenized version of foo.php we can see the levels of our sample script above.

Each Sniff in the standard has registered which tokens they are interested in being invoked to handle during the initialisation phase.

CodeSniffer then runs through each token in the file from beginning to end and calls all of the sniff process() method for sniffs that registered and interest in that token.

Finally all of the errors and warnings generated by those sniffs are organised into the desired report type and displayed.

So now we have an insight into how CodeSniffer works, in the next and final post in this series on CodeSniffer, we'll look and writing a new Sniff.