Expat Coder ≡ Brian Lyttle

SharePoint Intersection 2013 Talk

Wed, 30 Oct 2013 23:22:00 GMT

This was my first taste of presenting at a big name developer conference, and what a great event it turned out to be. If you attended my talk on Yammer API Development I hope you found it worthwhile. Please @mention me on the Yammer Developer Network with any follow-up questions.

There were some awesome from Elijah Manor, Bill Wagner, and many others. Co-locating the conference alongside Angle Brackets and the other Intersection conferences allowed me to dip into relevant talks from a range of speakers I normally wouldn't be able to see.

If you are interested in more of this, the next one will be held in Orlando in April.

Adding "Sign-in with Yammer" to ASP.NET MVC apps

Fri, 26 Jul 2013 04:07:00 GMT

One of the biggest pain points with enterprise web applications is user engagement stemming from another logon to remember, and mountain of profile fields to complete. Fortunately there are ways to improve this experience.

Depending on the situation you might be able to take advantage of Integrated Windows Authentication with access to Active Directory profiles, or have another identity service deployed. In my experience these can have a range of limitations including poor data quality, and a heavyweight programming model.

It would be nice to be able to implement something along the lines of Facebook Login. ASP.NET MVC 4.0 provides great support for social authentication providers, but out of the box it doesn’t provide support for any enterprise social networks. These building blocks are, however, an excellent foundation for implementing support for a Yammer provider.

Yammer is an enterprise social network (ESN) and at the core of the functionality is a user profile for each user within your organization. These profiles are controlled by users, and often have content that is more up-to-date than internal directories. In many cases customers are also using Directory Sync to update profile fields. I took some time to implement an OAuth 2.0 client which is compatible with Yammer, and I'm sharing the details below. This can be used in ASP.NET MVC sites, but it is something that you could modify to work with other types of .NET applications. There are however better choices for Windows Phone where an SDK is available.

Implementing the client

As ASP.NET MVC builds on functionality from DotNetOpenAuth the focus is on extending OAuth2Client as YammerClient. The complete implementation is available online, but the methods you need to implement are:

GetServiceLoginUrl() which is responsible for preparing the URL to redirect the user so that they can authorize your application.
QueryAccessToken() which executes a callback to Yammer in exchange for an OAuth access token.
GetUserData() which is responsible for querying profile data.

Something worth noting is that if you want to be able to make Yammer API calls, or silently check for profile updates, then you may need to modify GetUserData() to persist the token.

Using the client

Using the client in a new ASP.NET MVC Internet Application project is straightforward. Before going any further there are some steps you need to take to register an application on the Yammer site. These steps can be completed through the registered applications page, and detailed instructions are provided on the developer site. It is very important that you set a Redirect URI for your Yammer application. The format will be along the lines of http://localhost:31074/ for your development site in Visual Studio (adjust the port!), but you'll need to change it to something like https://www.myapp.com/authorize for production.

Back in Visual Studio you simply need to instantiate an instance of the YammerClient class, and register it in AuthConfig.cs:

public static void RegisterAuth()
        {
            const string appId = "Your Application ID";
            const string appSecret = "Your Application Secret";
            var client = new YammerClient(appId, appSecret);
            OAuthWebSecurity.RegisterClient(client, "Yammer", null);
        }

When your application is launched it'll offer the ability to sign in with Yammer and any other OAuth authentication clients that you have registered. It really is very slick!

A good next step is to modify your controllers and views to take advantage of the profile information beyond basic profile fields. That seems like good fodder for a future blog post.

Debugging problems with IE Security Zones

Wed, 24 Apr 2013 04:46:00 GMT

Internet Explorer has quite a complex security model compared to other browsers. One unique feature is the infamous Security Zone. Zones apply different policies to code based on the URL. By default, you have the following zones: Internet, Local Intranet, Trusted Sites, Restricted Sites, and My Computer.

Each of these zones has a slightly different configuration depending on the intended usage. For example, the Internet zone is designed to handle untrusted code from the internet. The settings are locked down tightly, and recent versions of IE run this code in a sandbox with Protected Mode.

Compare this to the Intranet zone which has a more relaxed configuration. If a URL is included in this zone it is considered to be trusted to a greater degree than one in the Internet zone. IE will even respond differently to authentication requests for sites in the Intranet zone to support Windows Authentication (see IWA, NTLM, Kerberos, and SPEGNO.)

Everything is fine and dandy when your application has code that only loads from URLs belonging to a single zone. However, things will start to go pear-shaped when you have code loading from different zones. The exact behaviour will depend on your application, but you'll often see inconsistent behaviour between environments. The explanation for this is often the application of group policy resulting in different connfigurations across machines.

So how do you go about debugging these issues? My approach is to start with the following:

Make sure the problem only happens in IE. Doing this will save you a lot of pain as debugging with the Chrome developer tools is often a more pleasant experience than using the IE tools.
Identify all of the domains hosting your code. Understand what zones contain your host page, and which ones contain other scripts or code that you are pulling in.
Temporarily move all of the code into a single zone. If you move all of your code into a single zone for testing you can confirm whether Protected Mode issues are at the heart of your problem. You may need to tweak some IE zone settings to get the features you need, but it's a good start.
Disable compatibility view. Sites in the Intranet zone use this feature by default, and it can cause problems. Temporarily turn it completely off for the Intranet Zone so that that things are consistent across zones.
Use IE Zone Analyzer. This tool allows you to review the configuration of zones in great detail, and help with bugs that only appear in certain environments. It is available for free from Microsoft.

At this point you'll be in a better position to debug specific code in IE, but how do you remediate the problem? Well you have to make a change somewhere. It is common for admins to add wildcards like *.domain.com to the Intranet zone. Naturally this will not discriminate between Intranet sites, and cause a lot of problems if you have modern code that doesn't run well under Compatibility View. Try to educate your admins on the appropriate configuration so that your environment will be more secure, and you'll also spend less time debugging!

Speeding up archive operations on OS X and Linux

Thu, 27 Sep 2012 23:46:00 GMT

I work with a lot of virtual machine images and other big data sets that needed to be archived, or moved around. If I run a command like tar -cjf archive.tar.bz2 directory it will churn away on a single core.

Given that all my machines have a lot of cores, it would be better if the tar command made use of all available cores. Thankfully someone created a tool called pigz (pronounced "pig-zee") that will allow the use of these CPU cores.

There are 2 parts to the setup for pigz: get the utility, and force tar to use it. On my Mac I grabbed pigz from brew:

brew install pigz

Then I added an alias so that tar uses this by default:

alias tar='tar --use-compress-program=pigz'

Remember that you might not want to use an alias if your profile travels with you to environments where the use of all cores could create problems.

Building Mobile Web Apps with jQM and Cordova on Azure

Wed, 16 May 2012 08:38:00 GMT

Thanks for attending my talk if you were able to make it over to Abington. I had to leave after my presentation so I didn't get a chance to see any of the other presentations, but the consensus appears to be that it was another great event. The slides are posted below for anyone that wants to reference them.

RedSnakePhilly 2012

Sat, 25 Feb 2012 23:27:00 GMT

I was asked to take a few photos for this event by fellow pythonista Tom Panzarella.

I think they turned out OK for the most part - although the EF 70-200 L saved my arse in the end. It was difficult shooting with the lighting setup at the venue, but I'll be back there to get some more practice.

All of the photos are over on Flickr and they were even linked to by Technically Philly!

Python interpreter tab completion on OS X

Sun, 09 Oct 2011 12:28:00 GMT

This is mainly for me own reference, but it is probably something that other people will find useful. If you attempt to use the rlcompleter module on the Mac you'll find that tab completion does not work correctly. Apparently this is because Apple ship a neutered version of readline.

To get completion working you have to use a different argument to readline.parse_and_bind(). The base example uses:

readline.parse_and_bind("tab: complete")

Change that line to:

readline.parse_and_bind ("bind ^I rl_complete")

Old but interesting news

Wed, 28 Sep 2011 09:38:00 GMT

I've spent a fairly big chunk of my professional career building public websites for a number of clients. One of the things that comes up again and again is search engine optimisation.

Surprisingly there is very little coverage of this is in business and marketing journals. Patrick Reid wanted to start raise awareness, and needed my technical help in preparing a paper. We finished it up earlier this year and it was published in the Journal of Strategic Management Education.

Corporate Communications in the FTSE 100: Evidence using Search Engines
JSME Vol 7: Issue 1, 2011
Patrick Reid (Greenwich School of Management, UK), Brian Lyttle (LiquidHub, Inc., USA)

Search engines have become an essential tool for information seekers on the world wide web. This makes it critical that company websites are optimised to achieve the highest possible ranking across the major search engines, in particular Google, ensuring maximum exposure and subsequent awareness. This paper identifies key factors in search engine optimisation and then analyses a sample of the largest 10 firms listed on the London Stock Exchange.

The findings reveal a lack of adoption of best practices which will impact on their search rankings. The paper also discusses tradeoffs inherent in search engine optimisation and draws out key implications for managers and academics.

The full paper is available from the Senate Hall site.

Scrapy dependency problems with lxml

Mon, 19 Sep 2011 03:31:00 GMT

Following the recent PhillyPUG meetup I was trying to install scrapy on an old MacBook Pro running OS X 10.6 (Snow Leopard) and ran into a number of problems with the lxml dependency. This is the parser used to extract data from pages that scrapy downloads so you are not going to get very far without it.

It seems that the compilation problems experienced when installing with pip result from an attempt to build a universal binary. If you have Xcode 4 installed then you lose some of this capability and need to make sure that the correct architecture is specified.

Architecture Fix

Setting the architecture is something you can do in your bash profile, executing it under a new bash ensures that the build script picks it up.

sudo bash
        export ARCHFLAGS='-arch i386 -arch x86_64'
        pip install lxml # test it
        pip install scrapy --upgrade # fix the failed scrapy install

Original Error

brianly$ sudo pip install lxml --upgrade
        Downloading/unpacking lxml
        Downloading lxml-2.3.tar.gz (3.2Mb): 3.2Mb downloaded
        Running setup.py egg_info for package lxml
          Building lxml version 2.3.
          Building without Cython.
          Using build configuration of libxslt 1.1.24
          warning: no previously-included files found matching '*.py'
        Installing collected packages: lxml
        Found existing installation: lxml 2.2.2
          Uninstalling lxml:
            Successfully uninstalled lxml
        Running setup.py install for lxml
          Building lxml version 2.3.
          Building without Cython.
          Using build configuration of libxslt 1.1.24
          building 'lxml.etree' extension
          gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch ppc -arch x86_64 -pipe -I/usr/include/libxml2 -I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c src/lxml/lxml.etree.c -o build/temp.macosx-10.6-universal-2.6/src/lxml/lxml.etree.o -w -flat_namespace
          /usr/libexec/gcc/powerpc-apple-darwin10/4.2.1/as: assembler (/usr/bin/../libexec/gcc/darwin/ppc/as or /usr/bin/../local/libexec/gcc/darwin/ppc/as) for architecture ppc not installed
          Installed assemblers are:
          /usr/bin/../libexec/gcc/darwin/x86_64/as for architecture x86_64
          /usr/bin/../libexec/gcc/darwin/i386/as for architecture i386
          src/lxml/lxml.etree.c:161594: fatal error: error writing to -: Broken pipe
          compilation terminated.
          lipo: can't open input file: /var/tmp//ccYr9GpX.out (No such file or directory)
          error: command 'gcc-4.2' failed with exit status 1
          Complete output from command /usr/bin/python -c "import setuptools;__file__='/Users/brianly/dev/github/pyconscrape/build/lxml/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /tmp/pip-axeEA7-record/install-record.txt:
          Building lxml version 2.3.
        
        Building without Cython.
        
        Using build configuration of libxslt 1.1.24
        
        running install
        
        running build
        
        running build_py
        
        creating build
        
        creating build/lib.macosx-10.6-universal-2.6
        
        creating build/lib.macosx-10.6-universal-2.6/lxml
        
        copying src/lxml/__init__.py -> build/lib.macosx-10.6-universal-2.6/lxml
        
        copying src/lxml/_elementpath.py -> build/lib.macosx-10.6-universal-2.6/lxml
        
        copying src/lxml/builder.py -> build/lib.macosx-10.6-universal-2.6/lxml
        
        copying src/lxml/cssselect.py -> build/lib.macosx-10.6-universal-2.6/lxml
        
        copying src/lxml/doctestcompare.py -> build/lib.macosx-10.6-universal-2.6/lxml
        
        copying src/lxml/ElementInclude.py -> build/lib.macosx-10.6-universal-2.6/lxml
        
        copying src/lxml/pyclasslookup.py -> build/lib.macosx-10.6-universal-2.6/lxml
        
        copying src/lxml/sax.py -> build/lib.macosx-10.6-universal-2.6/lxml
        
        copying src/lxml/usedoctest.py -> build/lib.macosx-10.6-universal-2.6/lxml
        
        creating build/lib.macosx-10.6-universal-2.6/lxml/html
        
        copying src/lxml/html/__init__.py -> build/lib.macosx-10.6-universal-2.6/lxml/html
        
        copying src/lxml/html/_dictmixin.py -> build/lib.macosx-10.6-universal-2.6/lxml/html
        
        copying src/lxml/html/_diffcommand.py -> build/lib.macosx-10.6-universal-2.6/lxml/html
        
        copying src/lxml/html/_html5builder.py -> build/lib.macosx-10.6-universal-2.6/lxml/html
        
        copying src/lxml/html/_setmixin.py -> build/lib.macosx-10.6-universal-2.6/lxml/html
        
        copying src/lxml/html/builder.py -> build/lib.macosx-10.6-universal-2.6/lxml/html
        
        copying src/lxml/html/clean.py -> build/lib.macosx-10.6-universal-2.6/lxml/html
        
        copying src/lxml/html/defs.py -> build/lib.macosx-10.6-universal-2.6/lxml/html
        
        copying src/lxml/html/diff.py -> build/lib.macosx-10.6-universal-2.6/lxml/html
        
        copying src/lxml/html/ElementSoup.py -> build/lib.macosx-10.6-universal-2.6/lxml/html
        
        copying src/lxml/html/formfill.py -> build/lib.macosx-10.6-universal-2.6/lxml/html
        
        copying src/lxml/html/html5parser.py -> build/lib.macosx-10.6-universal-2.6/lxml/html
        
        copying src/lxml/html/soupparser.py -> build/lib.macosx-10.6-universal-2.6/lxml/html
        
        copying src/lxml/html/usedoctest.py -> build/lib.macosx-10.6-universal-2.6/lxml/html
        
        creating build/lib.macosx-10.6-universal-2.6/lxml/isoschematron
        
        copying src/lxml/isoschematron/__init__.py -> build/lib.macosx-10.6-universal-2.6/lxml/isoschematron
        
        creating build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources
        
        creating build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/rng
        
        copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/rng
        
        creating build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/xsl
        
        copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/xsl
        
        copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/xsl
        
        creating build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
        
        copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
        
        copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
        
        copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
        
        copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
        
        copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
        
        copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
        
        running build_ext
        
        building 'lxml.etree' extension
        
        creating build/temp.macosx-10.6-universal-2.6
        
        creating build/temp.macosx-10.6-universal-2.6/src
        
        creating build/temp.macosx-10.6-universal-2.6/src/lxml
        
        gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch ppc -arch x86_64 -pipe -I/usr/include/libxml2 -I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c src/lxml/lxml.etree.c -o build/temp.macosx-10.6-universal-2.6/src/lxml/lxml.etree.o -w -flat_namespace
        
        /usr/libexec/gcc/powerpc-apple-darwin10/4.2.1/as: assembler (/usr/bin/../libexec/gcc/darwin/ppc/as or /usr/bin/../local/libexec/gcc/darwin/ppc/as) for architecture ppc not installed
        
        Installed assemblers are:
        
        /usr/bin/../libexec/gcc/darwin/x86_64/as for architecture x86_64
        
        /usr/bin/../libexec/gcc/darwin/i386/as for architecture i386
        
        src/lxml/lxml.etree.c:161594: fatal error: error writing to -: Broken pipe
        
        compilation terminated.
        
        lipo: can't open input file: /var/tmp//ccYr9GpX.out (No such file or directory)
        
        error: command 'gcc-4.2' failed with exit status 1
        
        ----------------------------------------
          Rolling back uninstall of lxml
        Exception:
        Traceback (most recent call last):
          File "/Library/Python/2.6/site-packages/pip-1.0.1-py2.6.egg/pip/basecommand.py", line 126, in main
            self.run(options, args)
          File "/Library/Python/2.6/site-packages/pip-1.0.1-py2.6.egg/pip/commands/install.py", line 228, in run
            requirement_set.install(install_options, global_options)
          File "/Library/Python/2.6/site-packages/pip-1.0.1-py2.6.egg/pip/req.py", line 1104, in install
            requirement.rollback_uninstall()
          File "/Library/Python/2.6/site-packages/pip-1.0.1-py2.6.egg/pip/req.py", line 487, in rollback_uninstall
            self.uninstalled.rollback()
          File "/Library/Python/2.6/site-packages/pip-1.0.1-py2.6.egg/pip/req.py", line 1417, in rollback
            pth.rollback()
        AttributeError: 'str' object has no attribute 'rollback'
        
        Storing complete log in /Users/brianly/.pip/pip.log

Deleting by ID (or primary key) with Fluent NHibernate

Sat, 30 Apr 2011 23:02:00 GMT

I wasn't a big fan of so-called Fluent Interfaces but my fondness is growing as I use Fluent NHibernate on an ASP.NET MVC project. When I started on my project I checked out the repository pattern example from Google Code to see how this was being implemented. The supplied repository interface provided the following interface:

public interface IRepository
        {
            T Get(object id);
            void Save(T value);
            void Update(T value);
            void Delete(T value);
            IList GetAll();
        }

Looks great apart from the fact that you need to pass an entity to the Delete() method, which would in turn result in iteration over a large number of objects if I had to do a lot of deletes. If I received an integer with an ID from an ASP.NET MVC controller I'd have to retrieve the object before sending it to the Delete() method. This is not very efficient.

To remedy this problem I added an additional Delete() method to the interface taking an object ID as the only parameter:

public void Delete(object id)
        {
            using (var session = sessionFactory.OpenSession())
            using (var transaction = session.BeginTransaction())
            {
                var queryString = string.Format("delete {0} where id = :id", typeof(TEntity));
                session.CreateQuery(queryString)
                       .SetParameter("id", id)
                       .ExecuteUpdate();
        
                transaction.Commit();
            }
        }

This is a very simple method that will generate more efficient SQL under the covers for deletion of objects from the database.

Custom ASP.NET MVC ActionResult classes

Fri, 22 Apr 2011 10:11:00 GMT

ASP.NET MVC provides a way for developers to extend and create custom ActionResults for controllers. Normally you return a view from a controller and the runtime goes off and hunts down the appropriate view based on the name of your action. This make sense for normal web pages but sometimes you need to build something in code or you don't want to write a view for it. In this case you can extend ActionResult, set a content type, and render the result to the output stream.

public OpenLayersActionResult MapData()
        {
            var myPoints = new List();
        
            var p = new MapPoint
                        {
                            Title = "Title for point on the map",
                            Icon = "marker.png",
                            IconSize = "21,25",
                            LatLon = "12.9715987,77.5945627",
                            Description = "This is the long description for a point on the map"
            };
        
            myPoints.Add(p);
        
            return new OpenLayersActionResult { Points = myPoints };
        }

Over on GitHub you can find some sample ActionResult classes including one that replicates the built-in JsonResult. The one that will probably be most useful to someone is the OpenLayersActionResult which simplifies generation of layer data. I'm digging through various projects and building out that library with other helper code from various projects so keep an eye out for more snippets.

Introduction to Google App Engine with Python

Mon, 11 Apr 2011 08:28:00 GMT

Thanks to everyone who attended my talk. I hope it was useful! If you haven't done so already please complete an evaluation form.

I've uploaded the presentation to Speaker Deck and you can download a PDF from my site too. The code sample I used during the presentation is available on BitBucket, and I'm going to be making various improvements to it over the coming weeks.

Installing Python Imaging on Windows 64-bit (AMD64) Python

Sun, 27 Mar 2011 03:31:00 GMT

If you want need a 64-bit version of this module for Google App Engine, or another project, you can run into a couple of issues. Sticking to 32-bit versions of Python and PIL avoids these issues so that may be the best direction for newbies.

The downloads on the PIL site are for 32-bit versions of Python which means that you hit runtime issues as soon as the Python script files attempt to load the 32-bit libraries into 64-bit Python. Annoyingly the installer is unaware of the architecture for the Python installation it finds and will leave you with a broken PIL install. Just uninstall it from Control Panel. My first thought was to try an build from the sources but I figured someone has run into this problem.

After some searching I found a site which offers pre-built versions of many Python modules for 64-bit architectures including PIL. If you are in any way concerned about performance, or security a better solution might be to build from source or use versions from a company offering supported versions.

I downloaded the version for Python 2.5 and then discovered that it would not install. What is interesting is that the 32-bit installer was able to find my 64-bit Python, but the 64-bit installer was unable to find it.

After some more searching it turns out that the installer takes it’s cue from a key in the registry and the Registry virtualization in Windows x64 was confusing it. To resolve the issue I had to export the “HKLM\SOFTWARE\Wow6432Node\Python\PythonCore\2.5” key, remove the “Wow6432Node\” string using a text editor, and re-import the key so that it was at “HKLM\SOFTWARE\Python\PythonCore\2.5”.

After doing this the 64-bit installer for PIL detected my Python installation and I was up and running.

Getting the most out of PyCon

Tue, 08 Mar 2011 12:33:00 GMT

Van Lindberg recently tweeted a link to some tips on how to attend a conference. The advice in that article is pretty sound for a business person looking to build up their network to find sales opportunities, but it doesn't cover any PyCon-specifics.

After a little thinking I've come up with my own list of tips for new PyCon attendees:

Attend some open space sessions

Open spaces are now happening at a lot of conferences but the PyCon ones have been the best I've seen. These are ad hoc discussions on a common topic with participants joining and leaving whenever they want. These sessions are a lot of fun and you'll make a lot of friends in the process. Even if you aren't the most talkative person, or know little about a subject, you are free to just hang out and listen.

Whilst there is a page on the PyCon website for these sessions, it is normally best to check the boards outside the rooms assigned to open space use. You should also monitor the Convore backchannel and #pycon on Twitter.

If you don't see any open space sessions related to your favourite topic just post a session in one of the available time slots. In almost all cases some likeminded individuals will turn up.

Scheduled talks are normally recorded for viewing later so I'd recommend that you try to decide ahead of time which talks are must see, and which ones could be skipped. That way you can make the most of the open space sessions as some of these will overlap with scheduled talks, especially on Saturday and Sunday.

Help the PyCon staff

Putting on a conference PyCon is an enormous effort. They are often looking for runners and other helpers. Spending a little time helping out introduces you to many interesting people and you'll probably learn a bit too about how conferences run.

Follow @pyconvolunteers, talk to the organisers, and lurk on the pycon-organizers mailing list.

Check out the lightning talks

These are very short talks on a variety of Python-related subjects scheduled at the beginning and end of conference days. If you have something to say to the community they are a good way to let people know about your project or passion. For everyone else they are a good way to find out about lesser known projects, and they are often quite humorous.

Go off track

PyCon like many other conferences has tracks with related content and there are 10 virtual tracks for 2011. These are really helpful for finding things that you are likely to be interested in, but sometimes it is good to check out something a little bit different.

In particular, veteran PyCon speakers like Alex Martelli, David Beazley, and Raymond Hettinger fill rooms each year and are always worth watching.

Chat with presenters

On the topic of presenters there are a lot of opportunities to talk to presenters at PyCon, either right after a talk, or later at the conference. If you have questions on their area of expertise or feedback on their talk it is worth taking some time to talk to them.

Spend time on things that are valuable to you

Sometimes you'll end up in a talk or open space session that isn't exactly what you expected. It is best if you go to something more to your taste than waste your time on a talk that is too basic or not as described. Sometimes this can be a little difficult if a session is packed with people, but you can do things to make your exit a little bit more comfortable.

First of all if you have an inkling that you are going to want to leave a talk don't pick a seat in the middle of a row. Instead sit somewhere that that has easier access to the exit at the back of the room. If there are video cameras in the talk you might not want to be "the dude that left in the middle of a talk". In reality you will likely not be noticed as the cameras are directed at the speaker.

Second, make sure to pack up your laptop and other things in your bag before exiting. It's kind of annoying to get clobbered with a laptop as someone exits past you. The same thing goes for anyone sitting in the row. Don't make unnecessary obstacles for anyone that gets out. Move your power cord and move closer to power strips rather than pulling them out of place.

Find a group for dinner in Atlanta

Watch the Convore and Twitter for information on groups who are going out for the evening in Atlanta. In many cases an open space session will continue over dinner!

Don't have too good a time at night

It is way too easy to have a good time on Friday and Saturday nights, but don't stay out too late or get too hammered. This seems like a bit silly but you need quite a bit of energy to make it all the way through a conference like PyCon.

I'm sure there are many things that I've missed here. Please post other tips on Twitter under the #pycon hash tag, on your blog, or post a comment below for others to read.

Using the WTL CListBox class

Fri, 04 Feb 2011 13:09:00 GMT

Working with Win32 listbox controls is fairly simple but the code can be quite convoluted and get tedious after a while. I decided to use the WTL library to simplify some listbox handling code but it wasn't initially obvious how to do this. Hopefully this blog post will help someone else understand the basics.

Manual message processing

The majority of Win32 programming involves working with messages of various types that are basically #defines in header files. The Listbox control responds to messages like these:

LB_GETCURSEL (get the selected item)
LB_DELETESTRING (delete a string)
LB_INSERTSTRING (add a string)

..but you also have to make sure you use the correct WPARAM and LPARAMs.

The resulting code for adding an item to a Listbox looks like this:

TCHAR szBuffer[80] = {0};
        UINT chars = GetDlgItemText(IDC_EDIT1, szBuffer, 80);
        SendDlgItemMessage(IDC_LIST1, LB_INSERTSTRING, (WPARAM) 0, (LPARAM)szBuffer);

As you can see this requires a lot of work that I’ve grown tired of with years of C# coding.

Simplifying with WTL

It is best to thinking of simplification here with a view to the long term. You are writing C++ so this is not as straightforward as WinForms or WPF.

First of all you need to create a member variable in your window class. I'm assuming you have installed the WTL AppWizard into Visual Studio and have created a Dialog application, but the steps are similar for other types of window. The code looks something like this:

class CMainDlg : public CDialogImpl<CMainDlg>, public CUpdateUI<CMainDlg>, public CMessageFilter, public CIdleHandler
        {
        private:
            CListBox m_ListBox;
        public:
            enum { IDD = IDD_MAINDLG };
        ....

As you can see we have private called m_ListBox that we can reference elsewhere. Next an instance of CListBox needs to be attached to a Listbox that has been added to your dialog resource file. You do this in the OnInitDialog() method:

LRESULT OnInitDialog(UINT /*uMsg*/, WPARAM /*wParam*/, LPARAM /*lParam*/, BOOL& /*bHandled*/)
        {
            m_ListBox.Attach(GetDlgItem(IDC_LIST1));
        ....

The IDC_LIST1 parameter is the name defined in my resource file and messages are now being routed between the instance variable and the actual control.

Now in the event handling code it is possible to call the AddString() method without having to worry about message types:

LRESULT OnAddItem(WORD /*wNotifyCode*/, WORD wID, HWND /*hWndCtl*/, BOOL& /*bHandled*/)
        {
            TCHAR szBuffer[80] = {0};
            UINT chars = GetDlgItemText(IDC_EDIT1, szBuffer, 80);
            m_ListBox.AddString(szBuffer);
            return 0;
        }

This is a rather naïve example as I’m not taking advantage of WTL for the Edit control, but that’ll be the subject of a future blog post.

Installing VMware ESX 4.1 from a USB flash drive on unsupported hardware

Mon, 24 Jan 2011 11:56:00 GMT

I recently replaced my aging Dell PowerEdge 830 server with a newer custom desktop and needed to migrate a number of old virtual machines. For the last few years I’ve been a happy user of VMware Server running atop Ubuntu Server and I figured it would be easiest to stick with that configuration.

So off I went with the usual Ubuntu 10.10 install and popped over to vmware.com to grab the latest virtualisation bits. It was then that I noticed that VMware Server (formerly GSX Server) is no longer being updated and users are recommended to upgrade to VMware ESXi.

From what I knew about the ESX product it had very specific requirements for installation, only supporting high end storage and a limited selection of NICs. It would seem that I needed to look at some of the Linux VM options like KVM or Xen, but it would be a pain to convert VMs over and it wouldn’t be straightforward to run these on my other computers.

Sticking with VMware

I went to Google and did a little research on installation of the latest VMware server version on unsupported hardware. VMware offer ESXi, a free Hypervisor, which is the base for their VSphere products. This is the recommended upgrade path for VMware Server users. You just register on their site and receive a free serial number.

Hypervisors like ESXi run directly on the hardware rather than on top of an operating system. This results in better performance and hardware utilisation.

A failed installation

The installer is supplied as an ISO image which is designed to work on supported hardware. As I found out, getting this to work well on unsupported hardware was problematic.

First of all I ran into a failure from what looks like a virtual disk driver (“vmkctl.HostCtlException Unable to load module /usr/lib/vmware/vkmod/vmfs3: Failure”). Whilst browsing many clueless forum responses I noticed a reply that mentioned Realtek network adaptors being a problem. Many motherboards including the Gigabyte GA-EP43T-USB3 in my machine use this brand of NIC, but VMware favour Intel cards. To find out if network card halted the installer I dropped into the BIOS setup and disabled the onboard network card.

On the next boot the ESXi installer let me pick a hard disk for the install but then blew up with a media error. It turns out that the installer has problems with your garden variety of DVD drive and people have more success with installation from USB flash drives.

I now had 2 problems: one with hardware (the NIC) and the other with software (the installer). I needed a new NIC and was able to pick up an Intel Pro/1000 PT Server card on Amazon with quick delivery. This was easy to install and it worked on first boot so I only had to get a USB installer working.

Creating a USB installation disk

My Google searches had taken me over to the vm-help.com forum, and I found a lot of useful custom installation and hardware/driver support information.

The first method I tried was a custom bash script called mkesxaio which enabled me to create an installation of VMware on a USB drive, but failed to create an installer. It was cool to be able to run ESXI directly from Flash memory as many people will want to do, but I wanted a solution that was safe from my cat. This bootable version did let me validate the performance of the new Intel NIC.

So it was back to Googling for other recommendations on creating a boot disk. I finally found a reasonable one by Ivo Beerens but I didn’t want to mess with Syslinux. Instead I used the weirdly named UNetbootin to install the VMware ESXi ISO image to my USB drive. This meant I could follow Ivo’s later steps to configure the drive.

The installer wants to validate some configuration that it expects on the CD so it is important to follow the directions around creation of a mod.tgz that sets the mtools_skip_check variable. Otherwise you have to drop to the shell each time you want to run the installation.

Kickstart files are used to automatically start the installation and I wasn’t able to find a configuration that let me boot the regular installer. It is very important that you have drives in the computer that do not have important data. The ks.cfg file contains a line specifying the drive to install to that is dependent on which SATA connection you are using. In short make sure the drive you want to use for the installation is the first drive, or modify the script.

At this point I had a USB installation disk that would automatically boot, provision ESXi, and then reboot to a fully working installation.

Final configuration

After installation I was able to access the console to verify network connectivity and enable some useful settings:

SSH remote access
Local troubleshooting mode

When I connected the IP address assigned to the machine I was presented with a the default server information and links to the vSphere client.

After a bit more work than I expected I now have a newer more powerful VMware environment for testing new tools, that will support newer guests like Windows Server 2008 R2. After a little bedding in time I’m going to migrate my existing VMs and get rid of the old Dell.

Production Debugging talk at Code Camp Philly

Mon, 11 Oct 2010 02:08:00 GMT

Thanks to everyone who attended my talk on Production Debugging. You can now download my slides and code samples from the talk. I have listed some questions and answers from the talk at the bottom.

Where can I get Reflector?

It is a free download on Red Gate's site. You don't have to buy the Visual Studio integration. When it starts up you are asked for the version of .NET to use. I always select the latest version as .NET backwards compatibility is good. Of course this can be changed later.

How do you stop someone from decompiling your code with Reflector?

The truth is that you cannot stop a determined hacker from getting at your code. Your best option is to make it harder by using an obfuscation tool such as PreEmptive Dotfuscator, Red Gate SmartAssembly, Remotesoft Salamander, Xheo Code Protection, or one of the many other .NET obfuscators.

You should be aware that obfuscators can make your life more difficult. One way they make code harder to read in Reflector is through method renaming. This can result in the stack traces from the obfuscated code being out of sync with your code.

Can I use ELMAH with Windows apps?

Not easily. ELMAH relies on various features of ASP.NET to do a lot of it's work. Some people have make their own modifications, but it not something that a beginning developer might want to attempt.

I wrote a sample ages ago called BugBack that works for WinForms. It is old and in need of some love but it is a starting point. If you want an off the shelf solution you should look at Red Gate SmartAssembly.

Multiple Monitor/Screen Information Example

Sun, 21 Mar 2010 01:59:00 GMT

Microsoft made it very simple to get information about the displays available to Windows with the Screen class in Windows Forms. Using this class you can identify the primary screen, and get properties like size from any display device.

For anyone developing an application which supports multiple monitors the Screen.Bounds property may be useful in finding screen offsets. If you have multiple monitors attached try the Get Current Screen Info button on your secondary display.

Download the C# sample code and binaries to explore further. Like all good API examples I have omitted error handling ;)

Looking forward to PyCon 2010!

Wed, 17 Feb 2010 13:04:00 GMT

On Thursday I'm making the yearly pilgrimage to PyCon in Atlanta. This will be my third year and I'm sure it'll be better than ever.

For me the real highlights of this conference are the legendary open space sessions. The level of interaction and learning at these really sets it apart from other conferences. Last year I attended a number of crackers including one on Cassandra and big data scalability with Jonathan Ellis. It's great to see that Jonathan is delivering a scheduled talk on database scalability, and I'm sure there will be a fair number of open space sessions dedicated to NoSQL databases.

Looking at the 2010 schedule these are my other top picks:

Deployment, development, packaging, and a little bit of the cloud (Ian Bicking)
Powerful Pythonic Patterns (Alex Martelli)
Understanding the Python GIL (David Beazley)
Mastering Team Play: Four powerful examples of composing Python tools (Raymond Hettinger)
Unladen Swallow: fewer coconuts, faster Python (Collin Winter)

I'll have my camera with me again this year so watch out for pics under the pycon and pycon2010 tags on Flickr.

Twitter to blog script

Mon, 25 May 2009 01:37:00 GMT

Based on an example provided with the Twitter library for Python I cobbled together the following script to add my latest tweets to this site. It's called from a cron job that I run on an occasional basis. My script linkifies hashtags and @username tokens in tweets so that you can see search results or user information.

Why did I not use one of the WordPress widgets? Well writing scripts like this is fun, and some widgets don't seem to play too well with my Thesis theme. One thing to note is that getting the shell script setup under some cron configurations can take a while if you aren't using it on a regular basis. It's operation is also different between Ubuntu Server and Joyent's Accelerator platform.

Up next: a similar script to process my latest bookmarks on Delicious.com.

Main script (tweets.py)

#!/usr/bin/python
        
        import codecs, re, getopt, sys, twitter
        
        TEMPLATE = """
        <li>
          <span class="twitter-text">%s</span>
          <span class="twitter-relative-created-at"><a href="http://twitter.com/%s/statuses/%s">Posted %s</a></span>
        </li>
        """
        
        def Usage():
          print 'Usage: %s [options] twitterid' % __file__
          print
          print '  This script fetches a users latest twitter update and stores'
          print '  the result in a file as an XHTML fragment'
          print
          print '  Options:'
          print '    --help -h : print this help'
          print '    --output : the output file [default: stdout]'
        
        
        def FetchTwitter(user, output):
          assert user
          statuses = twitter.Api().GetUserTimeline(user=user, count=7)
        
          xhtml = []
          for status in statuses:
              status.text = Linkify(status.text)
              xhtml.append(TEMPLATE % (status.text, status.user.screen_name, status.id, status.relative_created_at))
        
          if output:
            Save(''.join(xhtml), output)
          else:
            print ''.join(xhtml)
        
        def Linkify(tweet):
            tweet = re.sub(r'(\A|\s)@(\w+)', r'\1@\2', tweet)
            return re.sub(r'(\A|\s)#(\w+)', r'\1#\2', tweet)
        
        def Save(xhtml, output):
          out = codecs.open(output, mode='w', encoding='utf-8',
                            errors='xmlcharrefreplace')
          out.write(xhtml)
          out.close()
        
        def main():
          try:
            opts, args = getopt.gnu_getopt(sys.argv[1:], 'ho', ['help', 'output='])
          except getopt.GetoptError:
            Usage()
            sys.exit(2)
          try:
            user = args[0]
          except:
            Usage()
            sys.exit(2)
          output = None
          for o, a in opts:
            if o in ("-h", "--help"):
              Usage()
              sys.exit(2)
            if o in ("-o", "--output"):
              output = a
          FetchTwitter(user, output)
        
        if __name__ == "__main__":
          main()

Shell script executed as cron job (tweets.sh)

/usr/bin/python /path/to/tweets.py brianly --output /path/to/output/twittertimeline.htm

Keeping files in sync with Dropbox

Sun, 24 May 2009 06:25:00 GMT

When you use more than one computer on a daily basis, keeping files in sync between them is a constant problem. I'm familiar with tools like Subversion and Mercurial that make it relatively easy to keep code in sync between machines, but these require explicit actions that I often want to control. When it comes to photos, office documents, and other binary files that I don't want to actively manage these developer tools just get in my way. Thankfully a Y Combinator startup called Dropbox is trying to solve this problem and doing a pretty good job at it.

Dropbox is very simple to use. Just drop a file in a folder and it'll be uploaded to the central server. If your other computer is connected to Dropbox it'll be updated with the file almost immediately. It is worth noting that these updates are almost instant, even for large files. Whilst a sync tool like this is all very simple in theory, the execution is near flawless. I've tried out other tools in the past and none of them had seen the attention to detail that the Dropbox team made for their application. It get's better when you discover that Dropbox works just as well across Windows, Mac, and Linux.

I'm just about to shell out for one of their paid accounts so that I can keep my photos in sync between my Mac and Windows desktop. I'll post an update on my experience in a couple of months. I'm sure I'll try out some of the neat tricks detailed on their wiki.

Get fast computers for your developers

Mon, 09 Mar 2009 02:42:00 GMT

I've just been through an office move at a client site and packing it up got me thinking about the choices that companies make for their staff when it comes to computer hardware. Most users can benefit in some way from having faster computers, with bigger screens, and better input peripherals. Corporations have a tendency to make conservative choices and to pay above the market price for standard hardware. Whilst frustrating for normal users, this is an absolute performance killer for developers like me.

Compare the specifications of my home machine to my work computer:

Home		Work
2.4 GHZ Quad Core CPU (Intel Q6600)		2.0 GHZ Core Duo CPU
8 GB RAM		2 GB RAM
150 GB 10,000 RPM boot drive and several 1 TB 7,200 storage drives		160 GB 5,400 RPM boot drive
NVidia DirectX 10 graphics card		Intel discrete graphic card
Dual 24 inch TFT monitors		17 inch monitor
Windows Vista without Antivirus		Windows 2000 with Antivirus and other security tools.

My home configuration can be purchased for around $1200 today, and is many times more powerful than the work configuration. The ability to run multiple virtual machines as if they are real desktops gives a fantastic boost to my developer productivity. The screen real estate is beneficial for rapidly updating web pages, and the fast hard drives ensures that Windows keeps up with what I'm doing.

Since I don't run Antivirus at home I run as a user without administrative rights. If I need to test out new pieces of software from the Internet I use a Windows XP Virtual Machine running Windows OneCare Antivirus. Removing the performance overhead of antivirus means that I get the most performance out of my hardware whilst still staying safe. Did you know that simply running Windows with a standard user account can eliminate the threat of most Windows malware?

My advice to anyone buying hardware for development use is to buy the best performing hardware you can afford. Don't focus on any single component as skipping a dual screen configuration is not worth the price of a faster processor.

Essential Utility: htop

Mon, 09 Feb 2009 11:46:00 GMT

I've been rebuilding my Dell PowerEdge 830 server after a meltdown last week. This time I decided to go with Ubuntu Server 8.10 as the base operating system with Windows running under VMware Server 2.0.

Since this is a pretty small server, I'm always checking up on resource usage and top is a useful tool for this purpose. I find that I'd like to take actions based on what is happening in top and need to keep another shell open to issue commands. Wouldn't it be nice if there was a better version of top, that acted a little like the Windows Task Manager?

On Linux, there is an improved version of this tool called htop. This offers colour coding, CPU meters, and other nice things through the shell. Go check it out if you run Linux servers.

Weekly Links #2

Tue, 27 Jan 2009 12:18:00 GMT

PowerShell and svnadmin dump/load

Tue, 27 Jan 2009 12:10:00 GMT

Recently I was migrating several Subversion repositories from a Windows server over to a Solaris one and ran into a rather frustrating issue. Actually it's quite an interesting problem, but was frustrating due to the size of the repositories involved, and my fear that repositories were borked.

It all started with my decision to fire up PowerShell rather than cmd.exe to do the initial dump of the repostories. The following code was executed to dump a repository:

svnadmin dump repositorypath > repository.dump

All seems to go to plan but this is where a silent corruption of the dump file occurs. Not suspecting that the dump file was malformed, I copied repository.dump over to the Solaris box. I issued the standard commands to create a new repository and load the dump file:

svnadmin create repository/path
        svnadmin load repository/path < repository.dump

After executing the second command I got an error from svnadmin complaining about a malformed header on the dump file:

svnadmin: Malformed dumpfile header

My first instinct was to re-copy the dump file and try again, but I thought I should dump and load the repository on my Windows server. Lo and behold this fell over with the same error. At this point I was thinking something was really wrong with my SVN install and repositories, but then I stumbled on this newsgroup discussion.

It turns out that my problems were the result of how Windows PowerShell handles binary data that is piped on the command line. I really want to look into the details of this, but it would appear that the new found ability to pass .NET objects on the command line can mess with old school expectations of how the command line works.