Now that I've got the YAPC::NA and YAPC::Europe surveys done and dusted, I can now start looking at some of the changes I want to make to the survey system. Some I have already been planning, while others have been suggested by people who have taken the surveys.
Paint By Number Heart
The YAML file used to drive the surveys was originally designed to not reference any question numbers, with the code creating these on the fly to ensure no duplicates. To begin with this worked fine, but there are now such a variety of questions, including some that relate back to others, that knowing the question number ahead of time is useful. This is a fairly minor change, so shouldn't be too tricky to implement.
Take It To The Limit
I've had a few comments from people regarding the number of times they have attended a particular conference or workshop. Seeing as some have now been running for many years, if you've been to all of them, and can't remember how many that is, some have entered big numbers to imply they've been to all of them. As such, in future I'm going to add the feature that allows you to write 'ALL', as well as including the max number of occurances in the question label. Then if anyone does enter a rather large number or 'ALL' it'll get reset to the correct maximum.
Who Can It Be Now?
With the talk evaluations, a few speakers have asked whether I can tell them who wrote what in their evaluation results. Due to the nature of submitting everything anonymously I currently don't include any contact details with the responses. Having said that, there is no reason why anyone submitting an evaluation can't give their conscent to being identified, so that speakers can respond to the individual if they wish. The speakers concerned have had genuine reasons for wishing to contact individuals, so from next year there will be an extra tick box if you would like to be identified to the speaker. The default will still be anonymous submission, so it will be entirely the individual's choice as to whether they identify themselves or not.
Just Like You Imagined
I started writing a specification for the Survey YAML file during the Birmingham YAPC::Europe in 2006, but haven't kept it up to date with the changes that have been made in more recent times. I plan to complete this so that anyone reading the YAML files can make sense of them. It also might be useful for others to suggest new features.
P Machinery
I have the raw data from all the surveys since 2006 in SQL form. However, I want to make it more freely available and accessible, so others can analyse the data in different ways. Together with the raw data itself, this then needs the YAML file, the Survey specification and an example translation program to help understand how the data maps to the questions and what questions relate to each other. Essentially this just means me cleaning up the program I use to prepare the results and documenting everything.
Speak My Language
At the moment the survey site is written and presented in English. However, it's been long over due for the questions and text to be in a variety of languages. So although I don't speak other languages (at least not well enough to be competent in them), I'm hoping others will be willing to help out with some translations. Before I get to that stage though, I need to add support to allow for detecting and changing between languages.
Part of the reason for doing this is to allow YAPCs (and workshops) in other countries to take advantage of the surveys, if they wish to. At the moment I've only been working with the YAPC::Europe and YAPC::NA teams, as they have been the conferences I've attended, and are predominantly English language based. But these days I don't need to be there to run the surveys, and the Perl community pretty much covers the whole world, so why not accumulate knowledge from other events. At the very least, regular organising teams for workshops should then have the opportunity to get feedback to improve their events too.
Time Stand Still
It doesn't, but I often wish it would! A couple of people have commented that it takes such a long time between the end of the conference and the close of the surveys. This year I reduced it to 6 weeks, but it can then take an additional couple of weeks to process all the results, create graphs, prepare the website pages, write the documents for organisers and generate the emails for all the speakers. Much of it is automated now, but there are several tests and tweaks that need to happen to get it looking right. This will be improved further with the question numbering change, as I can fine tune the templates to particular questions.
As for improving the response rate, it often takes people some time to collect their thoughts. Typically within the first 2 weeks after the conference the bulk of responses are received. The last week sees another big increase in responses, when I send out the '1 week left' reminder. The weeks inbetween still have a steady trickle of responses, but it is slow. By way of example below are the number of responses received each day for the YAPC::Europe survey. The last 2 columns indicate the number of survey responses and the number of talk evaluation responses respectively:
Mon <-- evaluations opened Tue, 4th August 2009 - 27 Wed, 5th August 2009 36 234 <-- last day of conference Thu, 6th August 2009 34 119 Fri, 7th August 2009 31 101 <-- end of tutorials Sat, 8th August 2009 6 55 Sun, 9th August 2009 14 47 Mon, 10th August 2009 17 28 Tue, 11th August 2009 9 58 Wed, 12th August 2009 4 34 Thu, 13th August 2009 3 13 Fri, 14th August 2009 3 18 <-- end of week 1 Sat, 15th August 2009 1 3 Sun, 16th August 2009 2 11 Mon, 17th August 2009 4 2 Tue, 18th August 2009 2 - Wed, 19th August 2009 2 33 Thu, 20th August 2009 2 - Fri <-- end of week 2 Sat Sun Mon, 24th August 2009 1 7 Tue, 25th August 2009 1 - Wed Thu Fri <-- end of week 3 Sat Sun, 30th August 2009 1 6 Mon, 31st August 2009 1 - Tue Wed, 2nd September 2009 1 - Thu, 3rd September 2009 - 13 Fri <-- end of week 4 Sat Sun Mon Tue Wed Thu, 10th September 2009 1 - Fri, 11th September 2009 1 - <-- end of week 5 Sat Sun Mon Tue, 15th September 2009 23 48 <-- reminder sent out Wed, 16th September 2009 2 1 Thu, 17th September 2009 3 Fri, 18th September 2009 2 46 <-- end of week 6 Sat, 19th September 2009 2 8 Totals: 209 912
There is indeed a lull in the middle 3 weeks, so next year I do plan to try and reduce the availability of the surveys to at most 4 weeks. With the improvements to the evaluation and preparation code, it should then take about a week at most to review the data and then publish the results.
However, although my aim is to improve the turnaround of conference to results, there are a few other factors involved. All of my time taken to administer the surveys (before, during and after each conference) needs to accommodate my employer and family too. The conferences are typically around holiday season, so juggling everything to get the results done in a timely manner can be fun! Please bear this in mind if I don't fit into your personal schedule.
Release The Bats
It's always been my intention to release the code that runs the survey system as Open Source, however it means releasing two projects as Open Source, not one. The survey code is built on top of Labyrinth, a web management system I started developing in 2002 (2001 if you include the prototype). and is now used to run several CPAN Testers websites, as well as Birmingham.pm, GlousLUG and many other sites I run. Having a developer of 1 has meant it has taken a while for the code base to reach a stable state, which it has been now for a couple of years. It's not Catalyst or Jifty, although the core could potentially be abstracted and released as such. However, that's not what I'm going to release. Essentially I'll be releasing Labyrinth in two parts; the Perl library code and the supporting data files. On top of that I can then release the YAPC Survey code with the appropriate templates, plugins and additional data files.
Whether anyone then collaborates or not is another matter, but at least they'll have the chance to. People running smaller events will also have the ability to download, install and run the code themselves to administer their own surveys.
Pretending To See The Future
If you're an event organiser and think that the YAPC Survey system would be something you could use, please get in touch with any ideas you'd like to see featured. Also if you took any of the surveys this year, and think that they could be improved, whether it's simply rewording questions, adding questions, or some functionality that could be improved, please let me know.
I am very grateful for the support of all the conferences organisers, Eric Cholet for writing a new ACT API for me, and several speakers who have sent me personal thanks for supplying them with feedback to their talks, so it's obviously something that is being appreciated by the majority (though not all sadly). The split between the main survey and the talk/tutorial surveys seems to have worked well, and the majority of improvements so far have gone down well. There is still room for improvement, and hopefully the changes above will make for more feedback next year.
Re:Number of conferences
daxim on 2009-10-08T12:12:56
