meyerweb.com

Skip to: site navigation/presentation
Skip to: Thoughts From Eric

Archive: 'Projects' Category

Survey Analysis Service

During our analysis of the responses to the Web Design Survey, one of the things I thought seriously about doing was dumping the whole dataset into a database and building a web front end to query it.  Then I remembered that as back-end developer, I’m an excellent book author.  I know some MySQL and PHP, but I’m right in that sour spot of knowing enough to make the development process slow and error-prone due to my moderate but incomplete knowledge of the languages while not knowing enough to correctly design the project from the outset.  So I stuck to Excel and the like, which can be cumbersome but quickly learned.

I was a little sad, though.  I’d had the thought that if I built an interface to the survey results, it could be released publicly once we were done.  With such a tool, anyone could generate their own pivot tables without having to learn the process in Excel (or deal with Excel’s handling of enormous data files).  That seemed like a really good thing.

Well, the dataset is public now.  So how about one or more of you super-sharp developer types, the ones who didn’t check any of the boxes on the question about gaps in your back-end coding skills, doing what I could not?

The basic scope of the project would be to list the various data points (gender, ethnicity, age bracket, salary, geographic location, perception of bias, etc., etc., etc.) and let a user pick the two they want to analyze against.  So if someone wanted a table showing the breakdown of gender by ethnicity, they would pick one to go on the top and the other to go on the left.  The table generated would give those numbers.  I’d have it spit out raw numbers, but allowing the user to optionally get the results as percentages might be a nice touch.  Though then you’d have to let the user say which way the percentages are calculated: by column, or by row.

For extra deepness, one could also filter the results based on the value for a third data point.  With that sort of feature, one could get the breakdown of gender by ethnicity for only the EU respondents.  I might do it by letting the user click on a data point and then pick the specific filtering value via a dropdown.  Maybe three radio buttons: one for top, one for left, and one for filter.  Or, heck, do a whole Web 2.0 drag-n-drop interface.  That part’s not important.  What matters is giving anyone the ability to easily get numbers out of the massive dataset.

The only real challenge I can foresee is where questions allowed more than one answer, like the location of work question and the skill questions.  In the dataset, they’re just comma-separated value lists.  Those would need to somehow be broken out into subtables or Boolean columns or something.  The actual structure of the solution interests me a whole lot less than simply having one.

I’m quite sure this is the kind of thing a real programmer could create in about a day.  As I am not a real programmer any more, it would take me a month or four.  Let’s not wait.  Anyone out there able to take the idea and run with it?

Digging Into the Data

One of the practical reasons we released the anonymized data sets from the 2007 Web Design Survey was that we knew we couldn’t ask every possible question, let alone report on the results.  For that matter, we knew we wouldn’t even be able to come up with every possible question.  It’s one thing to approach this enormous mountain of data with a specific question in mind; those questions always seem obvious to the questioner.  In that case, there’s a clear path to the summit.  But we didn’t come at this with a specific angle in mind.  We just wanted to know what the profession looks like.  So we not only didn’t have a clear path to the summit, we didn’t even have a summit to reach.  Instead, we had thousands.  The tyranny of choice came down on our heads like a, well, like a falling mountain.

So the obvious choice was to release the data for others to analyze in search of their own summits, and I’m really glad to see people already doing so.  One gentleman is looking to produce an analysis of UK respondents, for example.  Others are asking specific questions and getting surprising results.

For example, Rebekah Ahrens grabbed a copy of the dataset and pulled out the answer to a straightforward question: what’s the gender distribution for the various age groups?  What she found was that almost without exception, the younger the age group, the smaller the percentage of women.  Here’s a chart showing the results she found in graphic form.

Wow.  What is causing that?  It’s a pronounced enough pattern that I initially wondered if it was somehow an artifact of the analysis method.  Several times during the authoring of the report I’d think I’d found some amazing and previously unsuspected trend… only to discover I’d divided some numbers by the wrong total, or charted column-wise when the table was row-oriented.  Mistakes along those lines.  They happen.  But I really don’t think that’s the case here.

Now, the really important question is why this pattern exists, and that’s where the data fail us—we can’t get the numbers to reveal all the forces that went into their collection.  There are any number of reasons why this pattern might exist.  I thought of three hypotheses in quick succession, and I’m sure there are many more of equal or greater plausibility.

  1. Younger women didn’t hear about the survey, and so didn’t take it.
  2. Women are losing interest in the field, instead heading into other career paths, and so those who have stuck with the field longer are more prevalent.
  3. Increasing margins of error at the low and high ends of the age spectrum reduces the confidence of the numbers to the point that we can’t draw any conclusions.

Remember, these are all hypotheses, any one of which could be true or not.  So how would we go about proving or disproving them?

  1. Conduct a survey of women in the field to see if they answered the survey, if they know others who did, the ages of themselves and those others, and so on.  Difficult to undertake, but not impossible.
  2. Ditto #1, although a possibly useful followup analysis would be to look at the gender distributions by longevity in the field and then cross-reference the two results.
  3. Get someone who is a statistician to figure out the likely margins of error to see if that might explain things.  I’d do it, but I have no idea how.  I would tend to be skeptical of this as an explanation given the clear trend, but I suppose it is possible.

I can, however, create the gender-by-longevity chart mentioned in #2 there.

Check it out: above six years’ longevity, women are consistently more represented (compared to the overall average) than they are below six years’ longevity.  The only exception is a spike at “1 year or less”.  Is that enough to explain the trend Rebekah spotted?  It doesn’t look like it to me, but then I’m not a statistician.  I also wonder a bit about the spikes at edges of the longevity spectrum.

I’m not trying to propose an explanation here, because I don’t have one.  I don’t even have an unsubstantiated belief as to what’s happening here.  I know just enough to know that I don’t know enough to know the answer.  What I’m saying is this: the great thing is that anyone can do this sort of analysis; and that even better, having done so, we can start to figure out what questions we need to be asking of ourselves and each other.

Analytical Breakdowns

As a member of the incredible team that’s worked so hard to create it, I couldn’t be more pleased to announce the publication of the findings of the 2007 Web Design Survey.

It’s amazing how many things this process broke.  Right from the outset, the enormity of the entire enterprise pushed all sorts of things past their limits.

The first thing the survey broke was our expectations.  I can only speak authoritatively for myself, but I think the general sense in the team was that we’d be really happy if the number of responses approached 10,000; I myself was expecting somewhere in the 7,000 – 8,000 range.  Instead, we got 32,831 responses.  So many people took the survey at once that it slowed down the Event Apart server, where the survey was hosted.

Although the survey software collected all that information with nary a hitch, getting it back out was another question.  Over the first few days, I was grabbing snapshots of the results as they came in.  That had to stop when an export request brought the entire server to a grinding halt.  Turns out we’d hit a limitation in the software’s code that made exports of tables above a certain size run very, very slowly.

When everything was finished and it was time to run the final export, the same problem bit us again: only this time, we couldn’t get the data out at all.  It was finally Mark Huot who extracted it by hacking the software a bit and running a custom-configured local server instance just to get it extracted in CSV format.

Which left us with a 35.8MB CSV file that needed to be cleaned a bit, as it had some funky characters in it that prevented a straight import into Excel (or anything else).  Have you ever tried to load a 35.8MB text file into a text editor?  It takes a while.  At one point, I honestly feared I’d locked up BBEdit.  And that was just to load it.  Imagine doing find-and-replace operations.  With grep.  (Now imagine constructing the grep expressions needed without gacking half the file.)

But eventually we got the file scrubbed clean and imported into Excel.  The end result was a 21.7MB Excel file.  Have you ever tried to work with a 21.7MB file in Excel?  It takes a while.  Hell, just hitting “Save” sometimes meant a three-minute wait.  And then there’s doing data analysis on a file that large.  Have you ever…?  Right, you get the idea.

The next thing that broke was our ability to process the data in meaningful ways.  I was able to pull some high-level numbers—what you might call surface analyses—for each of the questions.  For example, I could tell you the breakdown of titles, or genders, or education levels; in effect, all the charts in the final report’s introduction (that is, Figures i – xxviii).  At one point, I could even tell you what proportion of the respondents had 401(k)s.  But when it came to looking more in-depth, I was out of my depth.  Want to know the educational-level distribution of female part-time workers at non-profits?  I was lost.  A statistician I’m not.

So we hired two.  Jeffrey and I decided to use some of the proceeds from An Event Apart to retain two professional statistical consultants, Alan Brickman and Larry Yu, and we worked with them to identify interesting questions and patterns.  They did a superb job of not only breaking down the data for us, but also keeping our pattern-recognition impulses in check.  It’s tempting to look at a spike in numbers and invent reasons for its existence, and they reined us in more than once.

For example, and here’s a tiny peek into the results, we found that the higher a respondent’s salary, the more likely they are to participate in formal training (Fig. 9.6, p. 73, in case you want to go straight there).  But why?  Do they make more because they undergo more training; or undergo more training because they can afford it, or can afford to make their company pay for it?  Do not know, so cannot say.

Of course, it’s one thing to get a few hundred tables of numbers and a lengthy written summary from your consultants.  It’s quite another to turn it into a report that distills it all down to the essence, and looks good to boot.  And that’s where we started to heave real trouble.

I mean, sure, Excel can spit out charts like nobody’s business, but I have news: visually speaking, they really aren’t very good charts.  I know you’re shocked to hear this.  Perhaps under most circumstances that would be okay, but the charts that would go into our final report needed to look professional.  And by that, I mean designery professional.

As Head Chart Guy, I grappled with Excel (the Office X version) for weeks.  I learned more about Excel charting than I’d ever known, and still I could not get it to do what I wanted.  Right aligning the left labels on a chart?  The only way was to set the text rotation to something other than zero.  Then they’d all right-align, but also not be straight horizontal labels.  Plus they looked like crap that way, because there was no anti-aliasing happening on the text, or really anywhere else.  And so on.  We absolutely needed Excel to analyze the data, but its graphical output wasn’t what we needed.

So we looked around.  We considered PlotKit; we thought about pure CSS graphs; we even considered hand-drawing every last chart in Illustrator.  Eventually, we decided to give Numbers a try.  And immediately broke it.

Remember the big ol’ data files I mentioned earlier?  It turns out that Numbers was unable to even open them.  We couldn’t do any of our analysis in Numbers.  In hindsight, this was probably a good thing, because I don’t think it does anything like pivot tables, and those were absolutely fundamental to the whole process.  One personal benefit of this whole process for me is that I finally learned how to create pivot tables in Excel.  Strangely, it’s both easier and harder than you might expect.

So in the end, I produced needed data tables in Excel, copied them over to Numbers, and produced the charts there.  In the process, I learned way more about Numbers than I ever really wanted to know.  Jason and I could probably write a medium-sized book on all the things we learned, and learned to hate, about Numbers.  (Like, legends aren’t directly attached to their charts.  WTF?)

Which is not to say Numbers was a total disaster:  far from it.  All the charts you see in the report were created using Numbers, exported to PDF, and dropped into InDesign.  There were a lot of things about Numbers that made the process a lot easier than it would have been, and that very definitely helped us stay organized.  Of course, that just highlighted its limitations all the more harshly.  (You can’t change the separation distance between labels and the chart area?  WTF?)

And then there was the actual assembly, authoring, and proofreading.  Finding typos that inverted claims, mismatches between numbers, charts that were incomprehensible, summary tables that were calculated along the wrong axis.  All the innumerable errors that creep into a process, intensified and magnified by the unusually high information density of this particular project.  Late-stage changes of approach, starting over in places, realizing that things were not as we thought they were.  It’s the kind of thing that might lead one to break a web service.

And now it’s done.  It’s out.  We can rest.

It’s possible that you will not choose to rest, however.  I’m very proud to say that we’re giving the data back to the community that gave it to us in the first place.  Along with the report, we’re also providing anonymized copies of the complete survey data.  Every single one of those 32,831 responses, available in three formats, with only the identifying information stripped away.  If you want to fact-check our results, you can.  If you want to dig deeper, go for it.  If you want to investigate an area we passed over, please do.  It’s all there.  It’s all yours.  It always was yours—we just collected it and held onto it for a little while, and for that trust and honor, we thank you.

So: go read it and let us know what you think.  Tell us how we could do things better next year.  Show us what we missed by producing your own analyses.  And be ready for spring 2008, when we start the cycle anew.

Another Soul Lost to SketchUp

Hitting a link shared by Unstoppabot, who really needs to get around to fixing his feed linking policy (“View this site”?  Lame!), I was seized with another spasm of appreciation for the deliciousness that is SketchUp.

I did a moderate amount of 3D modeling back in the day.  The specific day in question would be the one where we all thought that images of rendered 3D models and, whenever possible, blobs of text were the absolute last word in Great Web Design.  Remember that?  Wasn’t it fantastic?  When every page title could be a bunch of extruded and beveled sans-serif letters viewed slightly from above, with the whole mess of text angled away from the observer?

Good times.

So anyway, while I was cranking out renderings of page title text and university logos, I also spent some time creating scenes of other stuff.  You can find some of the results if you dig deeply enough here on meyerweb, but that’s not my point.  What I’m trying to say is that I enjoy a bit o’ three dee more than most, and have some knowledge of how difficult it can be to construct models.

When I first heard about SketchUp, I was intrigued but didn’t really buy into all the hype.  It couldn’t be that easy, could it?  And then I watched someone using it—at An Event Apart, as it happens; and no, it wasn’t one of the attendees—and was captivated.  I downloaded the installer while I was sitting there, watching him create and modify shapes as easily as sketching them on paper.  And then I left it uninstalled, because I was afraid of what it would do to my free time.

A few days ago, I finally broke down.  I actually did have a legitimate reason to install and use it, a really good one, but of course I’d been waiting for any reasonable pretense to launch the .dmg and make with the modeling.  So I did.

Color me deeply impressed.  While you’re at it, add some heavy tints of addicted.  I started by modeling our kitchen, and now I want to do the whole frickin’ house.  I’m starting to eye local landmarks for recreation and contribution to the Warehouse and Google Earth.

I don’t have time for this.  I need help.  Stop me before I model again!

CSS Sculptor Released

I alluded yesterday to “backstage work”.  One of the things going on back there was my work with the folks at WebAssist to create a tool that allows both novice and experienced web developers to create and alter CSS-driven layouts.

The end result is Eric Meyer’s CSS Sculptor (v1.0.0), a Dreamweaver extension that gives the user tons of options and outputs pretty darned clean markup and CSS.  (Even if you’ve no interest at all in Dreamweaver tools, you should still follow that link.  How many chances are you going to have to see me throw the mëtäl hörns?)

Well you might wonder how much code I contributed to this software.  Well, okay, none.  What I did contribute was guidance on the interface and its organization; on what options to present, and how; on ways to handle things like print styles and the print CSS options; and on the CSS and markup that results, including the optional explanatory comments in the CSS.

One of the primary goals with this project was to create a tool that would expose as much CSS as possible to users without overwhelming them, and that would actually teach by dint of showing all the options.  One of the things we heard a lot in the beta test was users saying things like, “Oh, so that’s what that CSS thingy does! I’d always wondered.”  Which was exactly what we’d hoped to hear, along with “Hey, I’ve never heard of this CSS property before!”.  (We heard that one too.)

There are some things I expect will be improved in future releases, like shorthand value minimization—the simplest example of that being a condensation of 0 0 0 0 down to just plain 0.  We discussed including that feature but decided to postpone it for a variety of reasons, not least of which was working out the logic required to figure out when to minimize, which isn’t as simple a problem as you might first think.  There were a few other things we didn’t get in the initial release; such is the way of software.  We’ve got a list of potential features, of course, and are looking forward to hearing what users suggest.

As for what features did make it in, there’s a fairly large list, so it’s probably best to check out the Features page or take the product tour—with video commentary starring, you guessed it, yours truly.  So if you’ve ever wanted to see me greenscreened over screenshots and being intermittently goofy, then at long last your prayers have been answered!  (Or will be, once we get the load problems resolved.)

What makes this whole thing an especially interesting experience for me is that, for the first time in my life, I’m participating in an affiliate program.  Basically, what I earn from sales of CSS Sculptor depends on affiliate fees earned by referral links like this one here (and also earlier in the post).  That might sound like a rip-off, but it has the potential to be quite the opposite.  The affiliate cut is literally an order of magnitude greater than any reasonable per-unit royalty would be.

This compensation scheme (as they say in the UK) is actually an experiment on both sides: WebAssist has never really worked with an outside individual on a product like this, and so they honestly don’t know if the affiliate approach will pay better than a per-unit royalty, or worse.  So we’re going to try it out and see what happens.  Fortunately, I’m in a position that I feel I can afford to experiment like this, allowing both myself and WebAssist to find out what works best in the long run.

Thus, if you’re planning to buy CSS Sculptor or know someone who is, I’d be grateful if you either clicked through one of the links hereabouts or linked to the product using my affiliate URL.  I’ll have a link in the sidebar in the near future as well.  The sidebar’s due for an overhaul anyhow.

So that’s one of the things that’s claimed my time over the last few months, and I’m pretty excited that it’s seeing the light of release.  I’m even more excited about finding out what people think could be better about it so that we can improve what’s already a pretty darned nifty tool.  If I do say so myself.

San Francisco Schedule

Amongst all the travel, there’s been a metric ton of backstage work going on.  This is generally true of me these days, which is why posting has fallen off in 2007.  Unfortunately, it’s meant that I’ve been lax about keeping you folks up to date on what I’m up to—and also to keep you informed about An Event Apart, which is what accounts for most of that backstage work.

For example: last week, we announced publication of the complete schedule for AEA San Francisco, which will be 4-5 October 2007, and I didn’t say a word here.  I should have; honestly, it’s amazing.  I already want to see it.

I know, I say that every time, but it’s always true.  One of the things that makes me proudest about AEA, and that makes me continue to work hard on AEA, is that it fulfills one of the core requirements Jeffrey and I set out: to create the kind of event we’d want to attend.  I’m not satisfied with an AEA show unless I can look at it—and I mean all of it, from the schedule to all the organizational details that aren’t always obvious—and say, “I would pay money out of my own pocket to see this show”.

And so far, I’ve always been satisfied.

So we end the 2007 series with another great lineup and incredible set of talks in San Francisco, and it makes me proud all over again.  I hope you can be there to see it.

After Boston

Wow.

Just wow.

I’m back home and I still can’t believe how amazing An Event Apart Boston was for me and everyone with whom I talked.  I knew going in it was a great lineup of speakers covering great topics.  I knew that we had a completely kick-ass staff in place, and amazing volunteers to help us out.  I knew that we’d have great support from the venue.

I knew all that, and I was still overwhelmed and ecstatic at how things went.  At least on one level.  On another, thanks to the aforementioned kick-ass staff, things went so smoothly that I almost felt like I was a speaker at someone else’s conference.  I had so little to worry about that it was sometimes hard to remember that this was all happening because Jeffrey and I, over breakfast at Las Manitas in Austin, decided to take a chance and put on a show.  In a way, I had to prod myself just a little to remember to feel pride in what we’d accomplished.

What required no effort to feel was a deep sense of humility and awe that so many people had come to support what we did.  Over five hundred folks gathered in Boston, drawn by the same love of the web and pride in Doing Things Right that drives us.  I see the attendees at AEA as the craftsmen and women of the web.  Sure, there are shops mass-producing sites, the way a factory churns out cheap clocks.  That’s fine if you just want something to put on your nightstand.  But if you want an elegant, finely tuned work of art that you’d hang in a prominent place, a clock that is as much a point of pride as a timepiece—you find a craftsman.  And that’s who came to Boston.  That’s who comes to An Event Apart.

What amazed me even more was the overwhelming wave of positive feedback that we got.  Marci, our event manager, told me that in 25 years of event planning, she’s never seen attendees so happy.  So many people came up to me and Jeffrey and Marci just to say, “Thank you so much for doing this”.  They were thanking us, which seems entirely backwards.  I did thank each of them for coming to the event, but let me state it here for anyone I didn’t get to thank in person.  Thank you so much for coming to AEA and showing that you know creating the web is much more than churning out code, and that you take pride in being a craftsman.  Thank you for making the show so amazing.  Without you, it couldn’t have happened at all.

Now I’m looking forward to AEA Seattle twice as much as before, and I thought I was already maxed out on anticipation.

Again: wow.  Thank you, one and all.

Register for AEA Boston!

If you’ve been waiting to register for An Event Apart Boston, running March 26-27, the detailed schedule has been announced and the brand-new store has opened its doors.  Hie thee hence to sign up for two great days with nine amazing speakers in Boston’s historic Back Bay!  You’ll be glad you did.

(Pssst!  Just between us, you’ll be even more glad if you input the discount code AEAMEYE when you register.  It’ll give you a further $50 off the already-discounted Early Bird price, for a total savings of $150.  Add to that the discounted room rate at the conference hotel, and you could save something like $450 off the regular conference registration and room rates.)

The overwhelming feedback we got from 2006 attendees was that they wanted more, more, more.  More speakers, more insight, more time.  So that’s exactly what we’re doing with AEA Boston.  This is going to be the best Event Apart yet—with that speaker lineup, how could it not be?  Ethan Marcotte’s “Web Standards Stole My Truck”, Dan Cederholm’s “Interface Design Juggling”, Steve Krug’s “The Web Usability Diet”… and eight more sessions just as fascinating.  Furthermore, we’ll close out Day Two with live critiques of sites submitted by attendees, making recommendations on design, copy, code, and more.

One thing we’re not changing as we move from one day to two days is how we take care of attendees.  We’ll have delicious food for lunch and breaks both days, so you can relax and chat with your colleagues in attendance and not have to worry about finding a food court and running back to catch the afternoon sessions.  Our buddies at Media Temple will be throwing a first-night party for everyone so you can unwind and maybe do a little networking.  The fine folks at Adobe will have some great stuff to raffle off, with your registration as your raffle ticket.  In fact, it’ll be so great that they can’t even tell us what it is yet!  And those are just the high points.

Amazing speakers, a great location, great service, and big savings.  What more could you ask?

August 2014
SMTWTFS
July  
 12
3456789
10111213141516
17181920212223
24252627282930
31  

Archives

Feeds

Extras