Sewing pattern testing vs. scientific peer review

There is an interesting discussion going on in the GOMI craft forum right now (yes, I’m an avid GOMI lurker) in response to the recent pattern testing call and associated interview questions put out by Itch to Stitch.  Working in academia, I play an active role in the scientific peer review process.  Interestingly, this process is actually very similar to the sewing pattern testing process, with a few key differences.  Since I recently reviewed a paper and these ideas are fresh in my mind, I thought I’d share my completely unsolicited personal opinions.  🙂

(1) If you want to volunteer your time and money to pattern test, go ahead!  Really, I have no problem with people who enjoy this process.  I choose how to spend my time and money, and everyone else should have that same right.  Do whatever makes you happy.

(2) Personally, I have never and will never test a pattern.  If indie pattern companies start compensating testers with a fair wage and reimbursing them for all materials and supplies, it might be a different story.  But for now, when the company I would be helping will be making cold hard cash off of my time and money that are requested for free, it turns out to be a pretty raw deal for me.  This is economics, pure and simple.

Now, onto the peer review process.  During my recent paper review, I spent a substantial number of my normal working hours (and therefore my boss’s money that pays my salary) performing a service that provides no immediate benefit to either me or my boss.  The only immediate benefit is to a stranger, the author of the manuscript, another scientist that I have never and probably will never meet.  Not only do I receive no immediate benefit, but taking time to do the review actually put me behind on my other work.  Potentially, the author/stranger gets a big reward (the acceptance of their paper for publication), while my boss and I suffer a direct loss of time and money.  Sounds a bit like pattern testing, right?

Here’s the difference: By participating in the peer review process, I earn the right to ask other scientists, people who have never and may never meet me, to spend their time and money reviewing my manuscripts, potentially providing me with a big reward (publication) at the cost of their time and money.

In this scenario, everyone wins.  Everyone volunteers their time and money to perform a service for other scientists, and in return, everyone’s papers are improved by constructive feedback and eventually (hopefully) published, and everyone can boost their publication lists on their resumes, thereby earning promotions, fellowships, and awards…. in other words, more money.  I spend money on your review, and ultimately I’m paid back in salary increases and promotions.

Now, I’ll be the first to admit that the scientific peer review system does not operate as perfectly as my idyllic description above implies.  Like every community of human beings, we have issues.  However, in my experience, I personally feel that my time is being well compensated, and I am happy to participate in the process.

So, if I were the owner of an indie pattern company, and a fellow owner and I agreed to test each other’s latest patterns, I’d say this would be a fair exchange of time and money.  We’d both invest a bit of time, knowing that we’d both be making money off of our (hopefully improved) patterns.  Wonderful.

However, in the common scenario where an independent person is asked to spend their time and money with no reward, direct or indirect, or a direct reward (such as a copy of the final pattern) that is worth only a small fraction of the time and money they spent on testing the pattern, I’d say this is an unfair exchange.

If you want to do someone a favor out of the goodness of your heart, I applaud you.  This is one of the most beautiful aspects of the human spirit.  I volunteer my time for various causes (mostly STEM-related outreach and teaching) with absolutely no compensation other than the feeling that I am doing something good for another human being.  I do this outside of my normal work hours, during evenings and on weekends, and I often have to spend money on transportation to get to these events.  Likewise, if I ever find myself in a situation in which I am unable to get myself out of harm’s way, I would hope that someone would volunteer to help me.

All I ask is that we treat pattern testing as what it really is: volunteering.

The scientific peer review process is part of my job, and everyone is compensated by participating in the process.  I receive very selfish and tangible benefits: the publication of my papers, the beefing up of my personal resume, and the increased likelihood of earning more money.

If you receive a free pattern in exchange for pattern testing, this should not be considered fair compensation for your time and money.  This is a symbolic gesture thanking you for your time spent volunteering.

So, in conclusion, I offer the following suggestions:

(1) Let’s all acknowledge the unbalanced exchange of time and money involved in the current pattern testing system.  No one is claiming that fair compensation is taking place.  Everyone goes into this process with their eyes wide open, knowing what they will and won’t get out of it, and if all parties agree on the terms, so be it.  If pattern testers want to volunteer their time and money, that’s their prerogative.

(2) There is a difference between working and volunteering, and it is up to each individual to define their personal boundary between the two.  If you consider pattern testing to be work, then you should demand fair compensation.  If you consider it to be volunteering, you should expect nothing in return except a sincere thank you.

Personally, if someone is making money off my time and effort, I consider that to be work and expect to be fairly compensated.  If I teach a room full of high school kids how to identify mold species on a Saturday morning (true story), no one makes any money, and I feel satisfied that my time spent volunteering enriched the minds of future generations.

Now, I’m off to prepare a manuscript for submission to peer reviewers to claim my compensation for my recent review.  🙂

What do you think: do you consider pattern testing to be work or volunteering?  Should you be paid as a skilled contractor, or are you happy to help out an indie pattern company as a way of contributing to the sewing community?

Advertisements

40 thoughts on “Sewing pattern testing vs. scientific peer review

    • I think Victory Patterns and Grainline Studio does the testing in house …
      I know in one IG comment Victory Patterns asked one of her friend (which happened to be the owner of the Workroom) if she wanted to help test the new pattern (the friend was commenting how she was very excited about the new upcoming pattern)

      • I think testing in-house gives off a much more professional impression than testing by random bloggers. I know not all indie pattern companies can afford this though, so to each their own.

        As for Victory Patterns… don’t even get me started on them! I had a horrible experience with the one pattern I tried and swore off the entire company for good.

  1. I think the whole pattern testing thing as it currently stands is very iffy. However, I wouldn’t rule it out for myself, IF I would buy the pattern anyway, and felt that I’d personally benefit from a free pattern that I liked anyway. The pandering of much of the process does put me off though (I’m a long time GOMI lurker as well, love the discussions they have there)

    • This is a good point. If you think you’d buy the pattern and spend the time to sew it up anyway, pattern testing might be a decent trade-off. I know many sewists really enjoy being testers, so to each their own!

  2. Really interesting post, thank you! I’ve never done a pattern testing, and come to think of it, I probably wouldn’t for the same reasons you mentioned. Although I think there is an element of pride for some people who are asked to test the patterns, and also they get to immediately win the competition of who sewed that pattern first. Personal glory, perhaps? I do like the idea of indie pattern companies using other indie pattern companies for this job though. That makes a lot of sense.

    • I do think that some bloggers might also value the exposure, and potential new readers, of testing for a company with a lot of fans. You could wind up in a blog hop or round-up on the company’s site, which some people value.

      • All interesting points. If you’re a blogger who cares about being “picked” by popular bloggers, being the first to sew up a new pattern (which, let’s face it, is a great way to get people to read your posts), and getting exposure and click-throughs, then yes, pattern testing might be pretty beneficial to you in achieving these goals. Personally, I think that this really only makes sense if your blog is monetized, and all the extra page views will translate into additional advertiser dollars or purchases from your online shop. If you’re a hobby blogger, the only benefit (as far as I can tell) is winning a popularity contest, with no measurable impact on your real life. I stopped caring about being popular a long, LONG time ago, so for me, as a hobby blogger, spending time and money for click-throughs just isn’t worth it.

  3. Interesting thoughts. The definition of volunteering becomes a bit murky when there is no real ’cause’ as such. You volunteer to teach science – that’s great, and kids are enriched by your classes (and maybe you are too!). But how can you ‘volunteer’ for a for-profit company? Doesn’t that essentially become unpaid labour? In the past I have volunteered my time in retail at op shops, knowing that all the profits go to charity. But there is no way I’d do that for a regular retail shop, even if I liked their business and them as a person. Because ultimately they are making a profit off my back and that is scroogy and unfair.

    The other point to consider is: how can we be assured of the quality of independent patterns if they haven’t been properly tested? Let’s not kid ourselves that pattern testing by bloggers is anything but advertising. No matter how good a sewist they are, unless they have been trained in pattern testing there will be issues with the pattern that go unchecked. I’d feel a lot more comfortable if pattern companies actually shelled out the $$ for a professional pattern tester so there weren’t so many issues with fit and quality of drafting. I think Jenny of Cashmerette is going in the right direction: although she does the pattern testing rounds, her patterns are drafted professionally, so I would be more comfortable with the quality of their draft.

    • I have to agree that your average blogger is perhaps not the best pattern tester. The one time I tested a pattern, I gave feedback that the fabric width specified in the pattern isn’t really a width you find in the U.S. The designer told me that none of her several other testers mentioned this! Either they were all European, which I doubt, since she asked for testers on an English-language site, or nobody else noticed it? Yikes!

      • I agree on both points.

        (1) The idea of volunteering for a for-profit organization doesn’t make sense to me, unless the owner of the company is your mom or something.

        (2) The quality of feedback provided by pattern testers is probably very variable. (I don’t know because I’ve never participated in the process, but I’m guessing this is the case.) This brings up the point of professionalism, which was made quite convincingly by Stephanie Pearl-McPhee in the blog post linked to below. If you’re being paid to do something, it should be because you have some kind of expertise in that area, and thus your work will be worthy of your payment. Following on that, pay scales should be adjusted to reflect experience. If I charged a pattern company for my time spent testing a pattern, I’d definitely charge a lower rate than what I would earn in my primary field, in which I hold degrees and years of experience. But with that said, I’d definitely be charging *something*, US minimum wage at least, but probably more since I’m not a complete beginner.

  4. You have written about it well. I think pattern testing is work. Anything I sew that isn’t for myself I consider to be work, my second job. I am usually paid a price I am comfortable with… either a cup of coffee or a per hour rate. It just depends on the circumstances – who, what, when. The thing that bothers me about READING about pattern testing is seeing so many of the same top, or whatever, over and over again. It’s really interesting when two or three sewists make the same top. It’s amazing how different they can make their versions. When you start getting 6 to 10 versions of the same thing, I just don’t want to read anymore. Anyway this is all just my opinion, I hope everyone out there just has fun with their sewing.

    • Haha, good point Linda. The parade of pattern testers after an indie pattern release is nothing but marketing, pure and simple! I agree with you that it starts to get old pretty quickly, and it can even turn me off a pattern completely. Let me make my own decision without shoving it down my throat! I feel the same about sewalongs, especially ones that go on for weeks and weeks. Seeing the pattern name pop up in my reader over and over and over again is a huge turn-off. Plus, do people really need step-by-step photo tutorials on how to sew side seams? Tricky parts, ok, but photos for every single seam?

  5. Interesting ideas. I am also an academic and whilst there are numerous flaws with the peer review process I had never drawn the comparison with pattern testing. I think you have hit the nail on the head! At first glance there seem to be some similarities but in fact the lack of reciprocal benefit in pattern testing is shocking.
    I have never tested a pattern before, I have been offered a completed pattern in exchange for a review, which potentially seems like a fairer deal.
    Whilst there are problems, I can understand why some smaller companies do it and if there are people willing to offer their time and money then I guess that’s up to them.
    H.

    • Good point Helen – I agree that a free pattern in exchange for a review is at least a better deal than pattern testing, since in theory you could say whatever you want in the review, positive or negative. I suspect most reviews are biased positive despite the ubiquitous claims of “all opinions my own”, but it is what it is.

      Congrats to you for being both an academic and a functioning human being. I aspire to this. 🙂

  6. This was a really interesting post. I guess it depends on your perspective on testing, I only volunteer to test patterns that fit with my personal style and that I would be happy to receive a copy of the final pattern at the end. I feel like indie designers are quite upfront about it being a volunteer role and are more transparent about volunteers helping them create a product that the sewing community benefits from but that has no individual benefit. Whereas the quidproquo relationship in scientific peer review seems, at times, more fraught and not transparent about the fact that often we dedicate much more time to reviewing papers than we while receive in return. Currently the sewing community seems to be running well on kindness and goodwill as well as lots of skills and hard work and I hope it will continue to work like this.

    • Surprisingly, the sewing community does appear to run on a wave of (mostly) happy vibes. Kudos to everyone who contributes to this. I’m glad to hear that both pattern companies and testers are indeed acknowledging that testing is a volunteer activity Like I mentioned above, if all parties agree on the terms, then good for them. If sewing really is all about hugs and rainbows, then maybe I should stop complaining. 🙂

      Yes, there are many issues with the peer review system, including the skewed balance of quality reviews provided vs quality reviews received. I’m not sure what the solution is here, since you can’t force people to spend time if they don’t have it or, worse, don’t care. All I can do is continue to provide reviews that I think are thoughtfully prepared and hope that, by reading them, others will do the same.

  7. I think that there is an issue with work being done for free in that it devalues the work and therefore has a detrimental impact on people who are trying to make a living doing that work. I think that this is a particular issue in the arts where people are asked to work for free for ‘the exposure’. The point is made particularly well by Stephanie Pearl-McPhee in the blog post below.

    http://www.yarnharlot.ca/blog/archives/2011/12/06/unexpectedly_controversial.html

  8. I completely agree with you. After some pattern testing experiences I’ve had, it’s been made clear that testing needs more professional oversight. Amateurs (myself included!) who don’t know a thing about pattern drafting, just aren’t equipped to critique a pattern. If designers want to give away free patterns in return for a review (after being properly tested) well that’s a whole different thing. Promotion in return for promotion would be a mutually beneficial relationship… to those who need/want the blog traffic, anyway. I used to be more involved in pattern testing, but anymore it’s a rare thing. The last time I tested was for Megan Nielsen’s kids line, and I received a finished copy of a bundle of patterns, which dollar for dollar, would have come close to covering my labor and expenses. Anyway, I have many thoughts on this topic… thanks for laying it out so eloquently!

    • Interesting! I’m curious to hear more about your experiences testing and issues that have come up. Are testers reluctant to provide negative feedback, not putting enough time or thought into the process, or simply missing things due to lack of knowledge/experience? It does sound like a bundle of patterns is a better trade-off than just one, so I applaud Megan Nielsen for at least offering a more reasonable package. It would be nice if the industry shifted in this direction as a whole.

      • I do think there is a natural reluctance to give negative feedback. Many people tend to expect that what they’re receiving is basically a finished product. That said, I never overlooked errors for the sake of someone’s pride, and I’ve always found that the designers were very receptive to constructive criticism, and were more than happy to correct errors. There was one instance where the pattern I was testing had a rather major pattern error, and it wasn’t caught by any of the other testers. The designer was very grateful I found the mistake, but I found it so surprising that it was over looked by others! It just showed me that doing testing with the general sewing population probably isn’t a good idea. Had an experienced pattern maker sewn up a test, the issue would have been flagged immediately. I get that people are willing to test for the sake of “giving back to the sewing community” but by and large, I think community testing is doing no one favors.

        • Very interesting, thanks for sharing. I often feel the same way when reviewing papers, asking myself “how did no one else catch this??” At least I’m glad to hear that pattern designers are receptive to constructive criticism. When you’re a professional, evaluating criticism is such a critical step in improving your business, whether your business is sewing or anything else.

  9. The only time I agreed to test a pattern I got halfway through, finding several major flaws, and finally giving up on the final product. It took 3 lines of stitching for me to realize it was Not going to end well and it wouldn’t be to my style. I told the pattern maker the issues I had found and that it wasn’t my style.
    The problem was that indeed, I lost time, and a lot of fabric that I had cut before realizing the worst of the issues!
    Overall, I’d rather test for someone who is attempting to see if their pattern fits a different body type than attempt to see if it’s drafted well in general…

    • Ah, this sounds like a pattern testing disaster, and exactly why (in my opinion) testers should be compensated. In this scenario, you wound up with nothing to show for your efforts, except perhaps frustration. I’m sorry to hear this happened to you… and sorry that you wasted good fabric!

    • Thanks Suzanne. This has been an ongoing and evolving conversation in the sewing community for a while now, and I hope people continue to talk about it.

  10. hmmm…as a scientist who participates in the peer review process, I’ve never thought of it as a purely (self-interested) economic endeavor, but more as a community service and one of my responsibilities as a member of the scientific community. You certainly have a point that we benefit indirectly from this, but I would add that one benefit is making sure high quality research is published in the journals that we read. Could you say the same about pattern testing? While I’d never argue that pattern testing is any sewers ‘responsibility’, it does seem like testing for brands that you respect and enjoy might help make a better final product (win-win). The major difference as I see it is that the indie designer/company can decide what if anything they want to do with your feedback. In the peer review process you have the power to control what gets published. And come to think of it, the journal publishers are the ones making money off our free labor!

    • Patricia, you bring up many good points! I completely agree that as scientists, it’s our responsibility to curate articles and ensure that good science gets published, which benefits everyone in the scientific community. This “curating for the good of the community” happens in pattern testing as well, but the main difference I see is that the peer review process is reciprocal, whereas testing by individual sewists is not. My time spent reviewing papers entitles me to ask other scientists to review mine. So, the community benefits, but I personally benefit as well.

      As for journals making sh*tloads of money publishing our work that they had to spend absolutely no time or money to create, not to mention editors that coordinate the review process for free… well… that is an entirely different issue. I tend to file this away as “the cost of doing business” rather than let it drive me crazy with rage.

  11. This really is such an interesting topic and I’ve loved reading through all of the comments. When I first started my blog I really wanted to get involved with pattern testing because I thought it would be a good way to get some people to actually read my blog. I tested for anyone I could and wound up making things that weren’t really my style and weren’t patterns that I ever would have bought. (I am also pretty sure that my blog traffic didn’t increase based on testing those patterns, either.) Now that I have had my blog a little longer and have my core 6 people who are reading it (haha!) I have realized a lot of it isn’t worth my precious sewing time. I do still test for a few designers if they approach me but it is because I know I like their patterns and would buy them anyway. I tested one of the Megan Nielsen kids patterns like Lisa G and really liked it that the testers were given the whole pattern collection instead of just the one pattern we tested. A couple of the other designers I have tested for have given etsy gift cards or fabric store gift cards in exchange for testing. It’s nice to get a little something besides a free pattern in exchange for my time/effort/fabric. One of my favorite pattern designers, who I have tested for twice, actually encourages the testers to make our garments out of muslin or an old sheet or fabric we don’t value too greatly. She is the only person I’ve tested for who has stressed this, which I think is really how it should always be done. She is obviously looking more for fit than a garment that can immediately be put up on a blog to promote her new pattern. I do like seeing how various pattern designers work and I know it is sort of like volunteer work but I enjoy it here and there.

    • Teri, thanks for your very informative comments! I know you’ve done a lot of testing and really enjoyed hearing your thoughts on this. I like the idea of giving gift cards in exchange for testing; I hadn’t heard this before. Similar to the bundle of patterns, at least the exchange becomes a little more valuable and quantifiable in that case. Also, I LOVE the idea of that particular designer encouraging testers to use muslin. To me, this drives home the point that the designer is actually looking for constructive feedback rather than free advertising. In this case, I think the testing process becomes even more well defined as “work” because you don’t even have a usable garment that you can use afterward.

      On another note, I think you’re right that many bloggers use pattern testing to build their readership. Blog posts about new patterns always get a lot of hits (at least, I notice myself clicking on them!), and if traffic is what you’re after, it’s not a bad way to get it.

  12. At this point I’m kindof ambivalent about pattern testing. I did it twice I think, because I liked the designers, but it rubs against my planning nature to drop everything and sew a pattern that I might not even know what it looks like. I let projects simmer for months normally. But what is interesting is not too long ago there was a big dust up about indie pattern companies not being “inclusive” in their pattern testing. Meaning the random everyday bloggers were jealous they never got asked to test. Which I can understand why, because for a while the exact same people were testing patterns for tons of different companies over and over again. It was annoying and boring. But to counter act this complaining, companies did open calls for pattern testers and the quality of testers likely decreased.
    Maybe we’d be more happy if we didn’t hear anything about the testing process or see the tester versions like at Colette and Grainline (not that they don’t have issues with their patterns). Those two I can’t remember using testers as advertising recently.
    Another idea is if pattern designers want quality feed back and advertising, maybe they should go to sewing teachers more, then offer to host classes or events at a reduced rate at their store/studio? I’m not that person, so I can’t say if that’s a fair trade.

    • Great comments Molly, thanks. I remember that pattern testing hoopla a while back, and it was really interesting to see the changes that resulted afterward. Like you mentioned, the diversity of pattern testers in round-up posts increased dramatically, and I also noticed many more disclosure statements appearing on posts featuring free fabric or patterns in exchange for a review. Overall I’m pleased with both of these things. The fact that the quality of tester feedback declined is a shame, but not surprising if the more diverse pool of testers is operating from a more diverse range of skill (which is to be expected, really). I agree with you that testing and advertising are separate goals, and I think they should be treated as such. Get professional advice about fit, and use bloggers as free advertising on a finished or nearly-finished product.

  13. Interesting post — I’m glad you’ve brought this up!
    I was open to pattern testing, and felt that the nature of the task definitely fell on the “volunteer” side of the spectrum, rather than “work.” I am not a professional. It’s your customer “testing” (really, reviewing) your product, not an expert in the field doing so.

    That said, I recently tested a pattern (which I liked the look of; no regrets!) for an indie designer new to me. I’d expected a ‘free’ copy of the pattern, which I’d thoroughly review, and that my finished product would be included in her “tester roundup blog post or two utilizing some of the photos” provided by us volunteers. I should have read that sentence in the offer email more closely.

    The pattern, I received. The recognition on the designer’s blog, I did not. It’s a little embarrassing, but it was such a bummer to excitedly scroll through her post of other testers’ photos, checking out everyone else’s take on the pattern, only to realize mine was nowhere on the page. I wonder how many other testers were excluded.

    I suppose I was naive to have thought that these types of “roundup” posts always included all the testers photos… (I was then further put off after I emailed the designer, in hopes she was going to do a second post of photos, only to receive zero response).

    Pardon me while I go realign my expectations! I’m still happy to test patterns for designs I really like, but now I’ve come to expect even less out of the arrangement.

  14. Oh my gosh, I’m so sorry to hear this happened to you, but on the other hand, it very nicely illustrates my point! Honestly, the worst part was the lack of response by the designer. Not responding to emails is unprofessional in almost every circumstance, regardless of the type of business. As an example, Colette is notorious for ignoring (and deleting) critical comments.

    My personal opinion on this issue you described is that the best looking photos make it into the round-ups, with little or no discretion given to the quality of sewing. Bonus points if you’re young and pretty. I’m not trying to sound cynical, just being honest about what I’ve observed. In this scenario, testers are used for free advertising, pure and simple.

    A great example of this is poorly sewn samples being featured in otherwise gorgeous professional photographs. Again, Colette comes to mind. Do people really not notice the shoddy sewing? I notice this in tester round-ups all the time too. Beautiful photos, less than stellar sewing. In my opinion, not a great way to advertise a sewing pattern!

  15. Love this. Perfectly valid arguments. Most of the testing I’ve been privy to has been for companies that have affiliate​ programs, where you receive a small percentage from customers who purchase the patterns via an affiliate link. Some of my favorite fabric shops have the same program in place. In those situations, I feel a lot more fairly compensated for my time and effort, but again, it’s up to me to spread the link and pictures around so that I can receive compensation. My personal end goal in testing patterns is to do so and gain recognition in the sewing community, thereby gaining followers and potential customers. It’s definitely a long game, not a short one. And I may very well be the only one who sees it that way. Lol.

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s